Navigating the future of work: Skill-driven recommendations for job transition pathways

Job security can never be taken for granted. Especially in these times of widespread change accelerated by COVID-19. As governments exercise wartime measures to ‘flatten the curve’, many workers are forced to transition to new careers. However, moving from one job to another can be difficult or unfeasible when the skills gap is too large. 

Herein lies the main challenge to navigating the ‘Future of Work’: how do we efficiently transition workers at scale to secure high-quality jobs that meet new labour market demands?

Successful job transitions typically involve workers leveraging their existing skills and developing new skills to meet the demands of their target occupation. Developing the ‘right’ skills to bridge this gap is essential. This problem has motivated us to apply machine learning to labour market data and build a “job transitions recommender system”. By using real-time job ads data from Burning Glass Technologies, combined with employment statistics and household survey data, we’ve built a system that not only recommends career pathways but also the precise skills an individual should develop to help them get there. Our system is proven to be highly accurate, validated against historic job transitions.

Skills space

We started from the “ground up” by first measuring the “similarity” between individual skills, which are the underlying capabilities that enable workers to do their jobs. Each marker in the figure below represents a skill and the colours are clusters of similar skills. The closer the skills are together, the more similar and important they are when combined with each other in the same job. 

Source: supplied

The skills map above then forms the basis for measuring the similarity between occupations. Next, we estimate the similarity between sets of skills to identify how “close” professions are to each given their skill demands. The figure below visualises the similarity between Australian occupations in 2018. Each marker is an individual occupation, and the colours depict automation risk over the coming two decades (blue shows low risk and red shows high risk). Visibly, similar occupations are grouped closely together, with medical and highly skilled occupations having the lowest automation risk.

Source: supplied

The final step is to combine the measure of occupation similarity with a range of other labour market variables to build a Job Transitions Recommender System. Our system uses state-of-the-art Machine Learning techniques to ‘learn’ from historic job transitions and accurately predict job movements in the future. Not only does it achieve high levels of accuracy (76%), but it also accounts for asymmetries between job transitions. For example, on average, it would be more difficult for a ‘Janitor’ to become a ‘Lawyer’ than the other way around. This is because there’s an asymmetry of education, experience, and skills. 

Transitions map

The figure below selects 20 occupations and shows the probability of transitioning between each pair of them. We call this the Transitions Map. The way to read the map is the following: given a cell, the likelihood of switching careers from the ‘source’ occupation (the column) to the ‘target’ occupation (the row) is given by the shade of blue. Dark blue represents higher transition probabilities, and lighter blue shows lower probabilities. As observed, people tend to stick in the same profession even when changing jobs. There are evident asymmetries between occupation pairs. For example, a ‘Finance Manager’ has a higher probability of becoming an ‘Accounting Clerk’ rather than the reverse direction. And it’s generally easier to move to some occupations (for example, ‘Bar Attendants and Baristas’) than others (‘Life Scientists’). The dendrogram (the lines on the left and top of the chart) groups occupations based on their ease of transition. There is a clear divide between service-oriented professions and manual labour occupations.

Artificial intelligence-powered job recommendations

Sometimes, the new career requires developing new skills, but which ones? The AI system can help identify those. We built a Job Transitions Recommender System to make recommendations for both jobs and skills. We exemplify how tot system worlds for ‘Domestic Cleaners’, an occupation that experienced a severe employment contraction during COVID-19 in Australia.

First, we use the Transitions Map to visualise the occupations easiest to transition to. We also overlay occupations by their ‘essential’ vs ‘non-essential’ status during the COVID-19 crisis — blue markers indicate ‘essential’ occupations which can continue to operate during lockdowns, and red markers show the ‘non-essential’. We identify top-recommended occupations, as seen on the right side of the flow diagram (bottom half of the image), sorted in descending order by transition probability. Segment widths show the number of openings available for each occupation. The segment colours represent whether the demand has increased or decreased compared to the same period of 2019 (pre-COVID).

The first six transition recommendations for ‘Domestic Cleaners’ all experienced decreased demand, which is unsurprising since the government restricted ‘‘non-essential’ services in Australia during this period. However, the seventh recommendation, ‘Aged and Disabled Carers’, had significantly grown in demand during the beginning of the COVID-19 period, being classified as an ‘essential’ occupation. It is generally better to transition to high demand jobs, and we select ‘Aged and Disabled Carers’ as the target occupation for this example.

What skills to develop for new occupations

The Job Transitions Recommender System can also recommend skills that workers need to develop. There are time and resource constraints for developing new skills. So, workers need to prioritise which skills to develop.

We argue that a worker should invest in developing the skills most important to their new profession and which are most different from the skills they currently have. In the ‘Domestic Cleaner’ case in the figure above, the top-recommended skills to assist in transitioning to the ‘Aged and Disabled Carer’ occupation are specialised patient care skills, such as ‘Patient Hygiene Assistance’. 

On the other hand, the reasons not to develop a skill are when the skill is not important for the new occupation or when it is highly similar to their current occupation. The figure shows that while some ‘Aged and Disabled Carer’ jobs require ‘Business Analysis’ and ‘Finance’ skills, these skills are of low importance for the ‘Aged and Disabled Carer’ occupation, so they should not be prioritised. Similarly, skills such as ‘Ironing’ and ‘Laundry’ are required by ‘Aged and Disabled Carer’ jobs but it is likely that a ‘Domestic Cleaner’ already possesses these skills (or they can easily acquire them).

While the future of work remains unclear, change is inevitable. New technologies, economic crises, and other factors will continue to shift labour demands causing workers to move between jobs. If labour transitions occur efficiently, significant productivity and equity benefits arise at all levels of the labour market; if transitions are slow, or fail, significant costs are borne to both the State and the individual. Therefore, it’s in the interests of workers, firms, and governments that labour transitions are efficient and effective. Our research can help people to move between jobs efficiently and find high-quality work that supports their lives.

This article is curated from PLOS ONE.