Time for a new way to do continuous/future payments?

Not perhaps the best day in the year to publish this, but I'm the sort of person where if it doesn't get put down it gets lost.
So, I have a new debit card, not for any terrible reason, the old one just reached end-of-life. But while there is no drama behind it, there certainly is because of it.
This is the era of e-commerce, almost everyone takes card payments online and for the consumer the cheque is all but dead. Unfortunately this does mean every few years that a whole bunch of automatic, settlements fall over. Angry emails come buzzing in: this payment failed, that order canceled, if you don't pay soon your service will be canceled.
One interesting exception was TFL, they allowed my oyster auto-top-up to happen even though the payment didn't go through so long as I settled the outstanding amount with reasonable haste.
So what are the alternatives?

• Direct Debit; while used extensively in the public sector, utilities, and insurance, should there be a campaign to try and get more retailers to use this. Or is it the case that the framework agreement that it is based on provide too much risk?
• Third party solutions; These exist, for example Paypal allows you to set two methods of settling recurring payments. However these sorts of features come at a price and using value added payment services cost more.
• A new way? So is there something the banking industry can do?

What would this new thing look like, it would need to be easy and secure for consumers but not expensive for retailers to use. Is asking for both effectively asking for the moon on a stick? I imagine the biggest issue is can this be done without requiring expensive changes/upgrades to infrastructure on either side. Maybe a virtual card that can have a hundred year lifespan issued on a per site basis. Would it be a system of tokenisation that replaces actual cards in most places?

I really don't know, all I'm certain off is I'm not looking forward to going through the whole rigmarole in three years time.