Assessing Leadership Competence of Project Managers

Summary

This paper presents the concept of an assessment center that was applied as a method to evaluate employees in the department of software development in regard to the social skills necessary to ensure the production of high quality software in a team. The results of the assessment center are of multiple use. On an individual level, the assessment center provides important information for the further personal and professional development of the candidate. The assessment center and analysis have revealed that there is a category of large scale projects that places special demands on the project managers that are not necessarily provable by managing smaller projects successfully. Social skills such as dealing effectively with a variety of people in a context of various expectations, points of view and conflicts of interest are tested much more cost effectively in a simulation prior to managing a large scale project in reality. In addition, the assessment center turned out to be an important impulse impacting organizational development.
 

Introduction 

An assessment center is a procedure (not a location) that uses multiple techniques to evaluate employees for a variety of manpower purposes and decisions (Thornton and Byham, 1982). The effectiveness of the assessment center method has already been proven in practice on numerous occasions [...] and is used with particular success as a method for potential evaluation and management development (Seegers, 1989). This statement can be confirmed without hesitation by the positive results we have gathered by conducting numerous assessment centers for selecting potential managers at Allianz-Lebensversicherungs-AG.

While there is a long lasting tradition of evaluating managers systematically before they are promoted to a higher level of management, employees in the software department became project managers on a much more nebulous basis. The reason for this discrepancy is an historical one. In the early stages of software development important projects were always managed by managers on a hierarchical level. As the number of projects increased, responsibility of leading a project team spread to the level of employees on entry level.

An analysis of the, up to then, informal process of selection showed that there were three criteria for the decision to select a project manager (in order of priority): technical expertise, experience in managing a project and availability. In the last years, software projects have grown more complex and more costly. Meeting deadlines is more important than ever in a growing and liberated European and global market. Thus finding the correct answer to the question "Who is going to manage the project?? is a decisive aspect of success and failure in business. However, being an excellent project manager is not a matter of technical expertise but of being a good organizer and communicator. Tom DeMarco puts it this way: "If software development were just coding and debugging, we might get away with some kind of groupware, but the bread-and-butter activity of a software engineer is talking to another person. That?s how we spend nearly half our time" (DeMarco, 1995). However, it is not only the quality of the software to be developed that is at stake.
According to Ashkenas et al (1996), there are at least four types of boundaries that characterize most organizations and keep it from developing toward a high performance organization: vertical boundaries between levels and ranks of people; horizontal boundaries between functions and disciplines; external boundaries between the organization and its suppliers, customers, and regulators; and geographic boundaries between nations, cultures and markets. In my view, project teams, by virtue of consisting of employees belonging to different units of organization and / or levels of management, play an essential role in the process of making those boundaries more permeable. However, whether the work of a project team actually serves this function or not depends very much on the tenacity, ability and willingness of the project manager to integrate conflicting ideas and interests rather than simply attempting to circumvent conflicts and pleasing everyone. It is obvious that the demands on the social skills of a project manager cannot be underestimated and must not be neglected if quality software is to be developed and if the further development of the organizational culture is at stake.

Therefore, in order to systematically improve the selection process aimed at assigning the management of a large scale project to the right person the department of human resources and the department of software development decided to develop an assessment center to evaluate the social skills of potential project managers.
 

The Concept 

It was clear from the beginning, that neither technical or planning skills should be the focus of the assessment center since those skills are assessed on other occasions.

In order to assess the social skills of the candidates in realistic social situations, we decided to set up a scenario of an insurance company whose CEO has decided to enhance the presence of the company and the possibilities of communication with potential clients on the Internet. Although the CEO of the insurance company is convinced that this is a promising course to take, there are many problems to solve before being able to be virtually present and communicate with clients effectively on the Internet. There are technical problems to solve (e.g. aspects of secure data exchange with the client and the use of new programming techniques). There are problems on both social and cultural levels. Prior to this time, salespeople have personally sold insurance policies to their clients face to face. They are now afraid that someone is attempting to force them to use a new technology which they believe to be unnecessary. There are several competing ideas within the team regarding the best way to meet the challenge of the project.

Dimensions of Observation 
In order to obtain the dimensions of observation we first asked all senior managers and successful project managers in the software department to provide a list of all the demands on a manager of a complex project. In other words, we asked for a detailed job description focussing on social skills.
This task turned out to be an important impulse impacting organizational development: The managers were forced to become aware of their own expectations, discuss them with their colleagues and reach an agreement. This impulse was reinforced by having some of the senior managers observe the candidates in the assessment center and discuss and evaluate their observations afterwards.

The items on the list of social skills necessary for a project manager were categorized according to six dimensions (see Table 1):

Table 1: description of the observable dimensions

Exercises 
According to the given scenario we developed five role plays that simulate specific situations typical of a demanding project in order to assess the social skills of the candidate. An essential feature of the concept presented is that the five exercises are all part of one general scenario and move along a time line of about six months. However, in order to keep the simulation manageable and comparable among the candidates, we decided to keep the starting conditions in each role play identical - without regard to the actual course of action in the former role play.

In the following the content of the five role plays is roughly outlined. The reader should keep in mind that such role plays are easily described on a superficial level. However, developing a role play of high quality is very time consuming, especially if the role plays are interconnected. In order to obtain high quality we invested more than six months in the entire process of development. This included testing the role plays with volunteers and fine tuning several times before starting the first assessment center. The time and energy needed for shaping the characters, deciding what information is known to whom at what time as well as preparing all the papers and hand-outs necessary to make the simulation convincing and realistic should not be underestimated.

Criteria for determining the quality of the role plays were observability of the dimensions, validity of the content and the delicate balance between the complexity and the simplicity of the scenario that would allow all role players to act realistically according to the character they were to embody and the possible course of action as it develops dynamically during a specific role play.

We used trained role players, each of whom was prompted with general but also additional role-specific information to ensure a certain dynamic in a given role play. The role players were recruited among employees of the software department who were willing to volunteer.
In the following, the scenarios of the five role plays is described briefly.
 

First Meeting: "Getting started"
Scenario: the members of the team have never worked together as a team before. Each team member is aware of the outlines of the goal: enhancing the possibilities of interaction between the company and its clients on the Internet while fulfilling certain given requirements. The project manager must give the group the opportunity to warm up as a team. It is also important that he makes sure that there is a common understanding of the goal, a group sense of purpose and that existing misunderstandings are clarified by the end of the meeting. The project manager is also expected to work out a plan for further action with the members of the team.
 

One to one meeting with a member of the team: "more quality is needed"
Scenario: the project manager is repeatedly dissatisfied with the quality of work done by one of the team members. He needs to clarify the expectations in regard to quality and meeting deadlines. He is expected to establish a commitment for the goal of the project.
 

Team discussion: "The client wants additional software features"
Scenario: Although in the beginning of the project the sales department was rather skeptical of the new activities around the Internet, it has now discovered the new possibilities of selling insurance using the Internet as a tool. A senior manager of the sales department now demands that even more features of the software be developed. The new ideas are certainly interesting but the remaining resources in terms of time and budget do not allow for meeting most of the new demands. There are close and well established connections between the sales department and the CEO of the insurance company. The senior manager has strong expectations concerning a positive reaction of the project manager.
 

One to one meeting with a manager on a higher level of hierarchy: "more capacity is needed"
Scenario: the project has entered a phase in which much more programming capacity is needed. The technical problems concerning the security of data interchange are greater than expected. The project manager must ask for more capacity to program and debug the software needed.
 

Team discussion: "deadline in danger"
Scenario: Further unexpected problems arise. These can be partially traced to a decision made during an early phase of the project. At the time there was disagreement among the team members concerning this decision. Meeting the deadline seems almost impossible unless members of the team are willing to put a great deal of extra effort into solving the problems and postponing other obligations.

All the role plays are prepared in a way in which there is more than one possible outcome. By revealing only specific information and outlines of each role, the dynamic of each scenario remains uncertain and the outcome is dependent on the behavior of the project manager. The quality of the outcome depends on the communication skills of the candidate. Some important role specific information of a team member will only be revealed if the project manager is able to establish a climate of trust and commitment within the team. Some problem solving will only be possible if the project manager uses techniques of visualization, leading a group discussion and structures a group discussion according to the phases of gathering ideas, evaluating them and making a decision. In addition, some problems are only manageable if the project manager is successful in dealing with a conflict of interest among the team members.

Table 2 shows the matrix of dimensions to be evaluated (rows) and assessment exercises in which they are mainly observable (columns):

1. Exercise: First meeting: Getting started
2. Exercise: Meeting with a member of the team: More quality is needed
3. Exercise: Team discussion: additional software features
4. Exercise: First meeting: Meeting with a manager: More capacity is needed
5. Exercise: First meeting: Team discussion: Deadline in danger

Sequence of steps 

Figure 1 gives an overview of the sequence of steps to be taken by every candidate in order to make sure that the results were used for further personal and professional development.

The fact that the observers were involved in the process of developing the dimensions of observation and assessing the candidate as well as explaining the results to the candidate and their superiors proved to be very effective. This was due to the fact that it created a strong commitment to the concept of assessing potential project managers as a whole and not just to a small part of the process. Everyone was aware of the main purpose of the concept: diagnosing weaknesses and strengths for identifying potential project managers and identifying specific needs for further training of the candidate. It was the candidates responsibility to arrange the meetings in order to keep them in charge of the process as much as possible.
 

Discussion 

The purpose of the assessment center described above is twofold:

First, there was a need for a systematic evaluation of potential project managers in order to improve the selection process aimed at assigning the management of a large scale project to the right person.

Second, all candidates irrespective of their performance in the assessment center were given the opportunity to improve their skills by individually arranged training according to the thorough and intense feedback they had received.
Unexpectedly, the results of the assessment center served an additional purpose: The conference of observers in which all the observations of specific behavior were discussed and evaluated turned out to be an important impulse leading to further organizational development.

Observing the candidates in the exercises was, for the senior managers, like looking into a magnifying glass that reflects the organizational culture of the project management. The next day in the office they began a process of reflecting and improving the well implemented and, up to then, unquestioned project managing system resulting in a thorough revision of the entire process of systematic training of project managers. For example, the assessment center as described in this paper is now an essential component within the program of developing and assessing future large scale project managers. Another outcome is a regular and intensive coaching of the project managers by their senior managers or external coaches. Moreover, special workshops in which the participants learn how to design a kick-off meeting are now obligatory for new project managers.

The observers had to face the discrepancy between their expectations and what they actually observed in the exercises. They had expected more professional behavior from the candidates (e.g. dealing with conflicts of interest more actively and tenaciously as well as supporting the development of a group sense of purpose with more impetus). This assumption was made in light of the fact that taking part in this assessment center implied that they had previously successfully managed smaller projects and they indeed had been successful.

There are two possible explanations for the discrepancy:

First, the validity of the exercises is not high enough. This explanation leads us to the hypothesis that the candidates only failed because they had to act in a situation that did not accurately simulate typical situations during the course of a large scale project. However, in order to be selected as a participant in the assessment center, all of the candidates had been evaluated as skilled and experienced project managers who might be able to manage even larger projects. Were this not the case, their managers would not have selected them to take part in the assessment center.

However, the observers as well as the candidates confirmed the validity of the exercises even expressing their surprise at how well the simulation and the developing social dynamics in the role plays matched their experiences and mirrored typical situations during the course of a project. This fact might suffice to discard the first explanation at hand.

The second explanation came up on further analysis done by interviewing the candidates and their managers:

The results revealed a primary reason for the discrepancy between the observer?s evaluations and the evaluation of the candidates made by their managers prior to the assessment center. Whereas in the assessment center the candidates had to deal with people they had not worked with before, at home they were integrated in a social network they had built up over years in several projects. They had developed a manner of leading a project that was successful as long as the project team consisted of people who knew each other over an extended period of time and knew what to expect from each other.

In other words, it was not necessary for the project manager to attend to the sociology of the team to build up a team while dealing with demanding conflicts of interest. Considering only projects completed in this context, all candidates were successful project managers.
Comparing projects of different size, larger projects are characterized by a higher budget and a much tighter schedule. The sociology of such large scale projects is also different. The team consists of people from several different organization units. Politics also play a more important role. The conflicts of interest, which are likely to arise, require a project manager who is at least as skilled as a diplomat and not afraid to face conflicts when necessary.

As it happened, some of the candidates failed when confronted with the more demanding situation of an entirely new project team and the conflicts of interest that come along with a large scale project.

The assessment center and the analysis of the seemingly contradictory evaluations of the candidates made by the observers and the managers yielded an important result. Prior to the assessment center, senior managers operated on the premise that a given set of social skills was necessary for a project manager to successfully manage a project of any size. Additionally, managers believed that the successful management of a small project is automatic qualification for managing a larger project. In other words, promoting project managers by assigning to them more and more complex projects implied that the continuity of a growing budget size and complexity of the project is matched by a continuity of the demands on a given set of social skills. But moving along this supposed continuity turned out to be a costly misconception when the managers stumbled over a discontinuity: There was a critical degree of complexity and size (product profile, budget, number of organization units involved), beyond which a project manager would only be successful if he is able to meet the demands on additional social skills as outlined above.

So it occasionally happened that a project manager who had been successful in managing smaller projects failed to meet the demands of a large scale project.
 

Conclusion 

As a result of the assessment center and the analysis, it is now taken into account that there is a category of large scale projects that places special demands on the project manager?s social skills that are not necessarily provable by managing smaller projects successfully. Social skills such as dealing effectively with a variety of people in a context of various expectations, points of view and conflicts of interest are tested much more cost effectively in a simulation prior to managing a large scale project in reality. Furthermore, the outcome of the assessment center provides valuable information for determining the need for further training of the candidate and thus is an opportunity for systematic human resource development.

 

References 

Ashkenas, R; Ulrich, D.; Jick T.; Kerr, S.: The boundaryless organization - Breaking the chains of organizational structure. Jossey-Bass Publishers, San Francisco 1995.

DeMarco, T.: Why does software cost so much. Dorset House Publishing, New York 1995.

Seegers, J.J.J.L.: Assessment Centres for Identifying Long-term Potential and for Self-Development. In Organizations. In: Herriot P. (Ed.): Assessment and selection in organizations. John Wiley and Sons Ltd, Chichester 1989.

Thornton III, G.C. and Byham, W.C.: Assessment centers and managerial performance. Academic Press, New York and London 1982.


Download this Paper: assessment_center_for_project_managers.pdf