SÃO PAULO—The publicly traded units of troubled Brazilian investment bank BTG Pactual may face renewed downward pressure after the country´s market regulator rejected a request to extend a share buyback program, analysts say.

Brazil´s Comissão de Valores Mobiliários, or CVM, said late Friday that it decided not to permit BTG to exceed the regulatory 10% limit on share buybacks because that would lead to a reduction of the units’ free float and could possibly create artificial market conditions, according to a regulatory filing.

BTG´s former CEO André Esteves was arrested in November and later indicted on a charge of allegedly trying to obstruct an investigation into the bribery scandal at government-controlled Petróleo Brasileiro SA, or Petrobras.

Federal prosecutors allege that Mr. Esteves and a Brazilian senator had attempted to buy the silence of a key witness involved in the corruption investigation at the oil giant.

Both men have denied wrongdoing through their lawyers.

On the day of Mr. Esteves´arrest, BTG said that it would buy back up to 10% of the group´s units, which combine different classes of BTG´s shares, over the following 18 months. But the bank also asked regulators for permission to surpass the 10% limit and possibly acquire up to 41% of the outstanding units, according to the market regulator.

The price of the units plunged after the arrest of Mr. Esteves, who was later replaced as BTG´s CEO and chairman and relinquished control of the bank by swapping his voting shares for nonvoting ones.

Since November 24, the day before Mr. Esteves arrest, until last Friday, BTG Group´s market value has plunged 55%, to 12.9 billion Brazilian reais ($3.32 billion), according to financial information firm Economatica.

The bank acted to avoid an even deeper loss by buying back 8.6% of its outstanding stock from the day of Mr. Esteves´ arrest up until December 8, according to the CVM.

On Monday BTG´s units closed 3.35% higher, trading at 14.21 reais each.

BTG´s unit price “was supported this past week by the buybacks, which are now coming to an end,” said Max Felipe Bohm, analyst at Empiricus, an independent research company based in São Paulo. “This will increase the pressure for the bank to sell assets.”

To face a wave of withdrawals from its funds, BTG has been selling assets and has also secured a $1.6 billion credit line from Brazil´s deposit guarantee fund, or FGC.

Assets under management in BTG´s investment funds have been decreasing and people close to the bank expect withdrawals to continue as redemption dates for the investments are reached.

BTG Pactual on Monday said it was canceling 19.9 million traded units that were in its treasury as part of the buyback program.

The bank also reaffirmed its repurchase plans of buying up to 21 million units without surpassing the 10% legal limit.