Lloyds Banking Group says it has fixed problems that affected Halifax, Lloyds, Bank of Scotland and TSB customers using ATMs and debit cards.
Lloyds is the largest retail bank group in the UK with 30 million customers.
The chief executive of TSB, Paul Pester, said its customers' debit card transactions had been affected after problems with two out of seven IT servers.
He tweeted an apology and replied to some comments personally.
In a statement, Lloyds Banking Group said: "We apologise that earlier today, between 3pm and 6pm, some customers were unable to complete their debit card transactions.
"Although the majority of transactions were unaffected, we are very sorry for the inconvenience that this will have caused.
"At the same time, some customers encountered problems at approximately half of our 7,000 ATMs. This was resolved by 7.30pm, and all of our ATMs are now working."
When the problems first arose, TSB chief Mr Pester tweeted: "My apologies to TSB customers having problems with their cards. I'm working hard with my team now to try to fix the problems," he said, and he also took to Twitter to respond directly to unhappy customers.
He told the BBC two of the seven servers that process TSB debit card payments had suffered problems, which meant that about a quarter of TSB card transactions were affected.
Mr Pester later tweeted: "The cause of the problems at TSB is now fixed. It'll take a while to sort the backlog. Sorry customers may have problems for an hour or so."
TSB has about five million UK customers and is in the process of being split from Lloyds Banking Group.'Big nuisance'
Alexander Holland told the BBC he had just bought £90 of fuel at a service station when his card was declined.Continue reading the main story
Leanne Seaward, 29, from Verwood in Dorset, said she discovered she had problems when she went to pay for her weekly supermarket shop at about 16.15 GMT.
"It was a little embarrassing, I put my card in and it kept saying transaction void. I thought it was because I am in the process of switching banks, so assumed they might have closed my account without telling me.
"Luckily I had my husband with me so he was able to pay, but if I was getting petrol and on my own it could have been a completely different matter."
"They tried it several times and then suggested I used the cash point which I also tried but was declined," he said.
"What was I supposed to do? To make matters worse the police were in the service station and came over when I was explaining that there was nothing I could do but write them an I.O.U! How totally embarrassing."
Sam, a restaurant waiter, said he had served a couple of customers who were affected.
"Of course, they didn't know what the problem was and neither did we," he told the BBC. "All that happened was the card machines displayed transaction failed notices."
"It's a big nuisance and causes delays for other customers. It has a knock-on effect and banks need to be much more reliable if people are going to rely on their debit and credit cards to pay for services."
Not everyone who have tried to use their cards have had problems, and Lloyds said that online customers were unaffected.IT glitch
Prof Alan Woodward, a cyber security specialist at the University of Surrey, told the BBC: "It is quite unusual for two bits of hardware to fall over at the same time so I suspect it's something else that's happened.
"Quiet times like Sundays are when they do things like maintenance, so they'll do software upgrades or they might be doing back-ups. And it's those sort of things that tend to go wrong. So I suspect something like that has affected two machines at once.
"They might have been able one out of the cluster going down but two is probably what's causing the problem."
UK banks have suffered a series of IT problems over the past few years. In December, an estimated 750,000 customers at the Royal Bank of Scotland group were unable to use their credit and debit cards for several hours following an IT glitch.
In 2012, a major IT failure locked many RBS, NatWest and Ulster Bank customers out of their accounts for several days. The issue was particularly bad for Ulster Bank, with customers facing access problems for weeks.