Meta to start capturing employee mouse movements, keystrokes for AI training data

NEW YORK: Meta is installing new tracking software on U.S.-based employees’ computers to capture mouse movements, clicks and ​keystrokes for use in training its artificial intelligence models, part of a broad initiative to build AI agents that can perform work tasks autonomously, the company told staffers in ‌internal memos seen by Reuters.

The tool, called Model Capability Initiative (MCI), will run on work-related apps and websites and will also take occasional snapshots of the content on employees’ screens, according to one of the memos, posted by a staff AI research scientist on Tuesday in a channel for the company’s model-building Meta SuperIntelligence Labs team.

The purpose, according to the memo, was to improve the company’s AI models in areas where they struggle to replicate how humans interact with computers, like choosing from dropdown menus ​and using keyboard shortcuts.

“This is where all Meta employees can help our models get better simply by doing their daily work,” it said.

The Facebook and Instagram owner has been moving aggressively to ​integrate AI into its workflows and reshape its workforce around the technology, arguing it will make the company operate more efficiently.

Meta CTO Andrew Bosworth told employees ⁠in a separate memo shared on Monday that the company would step up internal data collection as part of those “AI for Work” efforts, now re-branded as Agent Transformation Accelerator (ATA).

“The vision we are building towards is ​one where our agents primarily do the work and our role is to direct, review and help them improve,” Bosworth said. The aim, he added, was for agents to “automatically see where we felt the need to intervene ​so they can be better next time.”

Bosworth did not explicitly spell out how those agents would be trained, but said Meta would be “rigorous” about “building up data and evals for all the types of interactions we have as we go about our work.”

Meta spokesperson Andy Stone acknowledged that the MCI data would be among the inputs.

AI WORKFORCE OVERHAUL

Stone said the data gathered via MCI would not be used for performance assessments or any other purpose besides model training and that safeguards were in ​place to protect “sensitive content,” without elaborating on which types of data would be excluded from collection.

“If we’re building agents to help people complete everyday tasks using computers, our models need real examples of how people ​actually use them — things like mouse movements, clicking buttons, and navigating dropdown menus,” said Stone.

The push to automate functions previously performed by human staffers reflects a broad pattern among major U.S. companies this year, especially in the tech sector.

AI ‌tools have captivated ⁠Silicon Valley with their ability to handle complex tasks like creating apps and organizing large volumes of data with limited human oversight, sparking a selloff in stocks of traditional software companies and inspiring some executives to plan extensive job cuts.

Meta is planning to lay off 10% of its workforce globally starting on May 20 and is eyeing additional large cuts later this year.

Amazon.com similarly has trimmed 30,000 corporate employees in recent months, representing nearly 10% of its white-collar workers, while in February the fintech company Block chopped nearly half of its staff.

Internally, Meta has been exhorting staffers to use AI agents for coding and other tasks, even if it slows them down in ​the short term. It has also been wiping ​out distinctions between certain job functions in favor ⁠of a new general-purpose job title called “AI builder.”

Last month, it created a new Applied AI (AAI) engineering team aimed at improving the coding capabilities of Meta’s AI models and using them to craft AI agents that can perform the bulk of the work to build, test and ship future products and infrastructure at Meta.

Meta started transferring “strong” ​software engineers into AAI earlier this month.

WHITE-COLLAR SURVEILLANCE CONCERNS

Computer logging and screenshotting technology have historically been used by companies to hunt for employee misconduct or non-work-related ​activities, said Ifeoma Ajunwa, a ⁠law professor at Yale University.

The move to log employees’ keystrokes takes the data-gathering goals a step further, she said, subjecting white-collar employees to a degree of real-time surveillance previously experienced only by delivery drivers and gig workers.

“On the U.S. side, federally, there is no limit on worker surveillance,” Ajunwa said, adding that state-level laws require at most that workers be broadly informed when employers are monitoring them.

European law would likely prohibit such monitoring, said Valerio De ⁠Stefano, a law ​professor at York University in Toronto who studies technology and comparative labor law.

In some countries, such as Italy, using electronic monitoring ​to track employee productivity is explicitly illegal, while in Germany, courts have held that employers can deploy keystroke logging only in exceptional circumstances, such as suspicion of a serious criminal offense.

Additionally, De Stefano said, the practice would likely be considered a violation of Europe’s ​General Data Protection Regulation.

More broadly, he said, awareness of employer surveillance shifts the balance of workplace power in the employer’s favor.