Amid wide public debate about the need for new online surveillance powers, he said: "The Prime Minister was right when he was saying last week we can't afford to have complete no-go areas.
"We cannot have no go areas in our communities where the police cannot go, because that just allows space room for the evil-doers to ply their trades.
"It is the same in the virtual world.
"If you allow areas which are completely impenetrable then you might feel comfortable that your communications are private and no one else can see them, but so are those who are trying to do you down and undermine your society."
He said the revelations by Edward Snowden, the former CIA contractor, had captured the public's attention because they had not been properly informed about the security services' online surveillance capabilities.
Snowden, who revealed top secret details about online monitoring, had "thrown a massive rock in the pool", said Sir John, but he admitted that the move had led to a public debate.
Sir John went on that Snowden "gained traction" because the public did not know that GCHQ, the Government listening post, and its US counterparts "could monitor traffic on the internet in the way that it could".
"It was certainly a great concern for me that the, if you like, the informal co-operation that worked well between most technology companies and communication companies and security services was broken by the Snowden revelations and has not been repaired," he said.
"These new developments in technology and in communications are vastly advantageous to our economies and to our way of life and to family cohesion.
"But if a technology companies allow to be developed areas which are simply impenetrable, you are inviting problems.
"We have to find a way as a society whereby the technology companies ... and those responsible for the security of our societies can work together so that the interests of both can be met with limited compromise.
"Now, I do not believe that there is a trade-off between security and privacy. I think they go together.
"If you have a society which evades and abuses privacy, then ultimately there will be a reaction against the damage to your security.
"If you do not have any security then all your basic freedoms are at threat.
"So we have to find a way of building the trust in governments and in technology companies that both the private uses that people justifiably expect to be able to have of these new technologies and the public goods are both being met at the same time.
"Now Snowden threw a massive rock in the pool. The ripples from that still have not died down, but it has provoked a debate on these very difficult issues.
"There needs to be some new compact between the technology companies and those who are responsible for security if we are not to see events like we saw in Paris last week and which we have seen also across in Yemen, in Nigeria, and so on become more and more features of our lives."
"We cannot afford for that to happen."
Sir John said there were so many terror plots ongoing that it was a certainty that the security services would not be able to foil them all.
"We are not saying that an attempted terror attack is highly likely, but that an attack actually getting through is highly likely," he told an audience of business leaders in London.
Sir John said while the majority of the British-based Muslims who had gone to fight in Syria would probably pose no security threat when they returned home, a "hard core" would almost certainly attempt to mount terror attacks.
"I think the great majority of them are probably mightily relieved to get back [to the UK], but there will be a hardened core who are absorbed into extremist ideology who will pose a real threat to us back here.
"The security community has done a fantastic job keeping threats at bay, but if I was to sit here and ask 'will the goalkeepers of the police and security services foil every single attempt to score a goal?', the answer is no.
"At some point a threat will get through," he said at the central London launch of the Edelman Trust Barometer, an international survey of public trust in institutions including the security services and other government institutions.
The UK survey found public trust in the intelligence agencies was far higher than for other official bodies.
It found 64 per cent said they trusted MI6 and 72 per cent trusted MI5, the domestic security agency, compared with a 43 per cent rating for government overall.
Sir John said the security agencies had begun to rebuilt public trust following concern over the "dodgy dossier" under Labour prime minister Tony Blair in 2003, which was later used to justify the invasion of Iraq on the grounds that Saddam Hussein's regime was claimed to have weapons of mass destruction.
Sir John said the British agencies had achieved the turn-around by being "skilful and clever rather than violent and thuggish", in a reference which will be widely interpreted as a veiled swipe at the CIA.
The American agency's involvement in torture was detailed in a report from the US Senate Intelligence Committee last month.
Spy fiction such as James Bond films also helped boost the reputation of the British agencies, Sir John admitted.
"Our favourable reputation in popular fiction, as we all know, helps on trust," he said.