Stevej on NBN: Brandis's stumbling and bumbling has given away more State secrets than Snowden.

Expanding on my previous comment that only ordinary citizens, no serious criminals or terrorists, will be caught by the mismanaged & mangled "thought bubble" of Brandis on metadata retention .

Brandis is a fool or knave, probably both. he's given away two important intelligence secrets, he's trying to secretly "backdoor" his draconian "Copyright Piracy" laws and he's covertly, possibly unconstitutionally, trying to push costs of Government Intelligence operations onto the private sector.

Viz:

He has deliberately revealed not one, but two, highly classified details about the SIGINT capability of ASIO, the AFP, our other LEO's and National Security agencies. This starkly contrasts with Morrison's tight-lipped silence about "On Water Activities". This is government that seems perplexed and conflicted about its job.

The exact nature of a rather critical and growing gap in the data available to intelligence agencies. Exactly the same gap that was exploited, en masse, in the UK riots.
Leaked critical data about the highly classified internal process ASIO etc uses in detecting and surveilling terrorist and suspected terrorist activity.

Tried to sneak past us his draconian and ham-fisted "illegal downloading" laws under the guise of anti-terrorist activities, via a "general crimes" catch-all.

The Source/Destination IP numbers catch Torrents "illegally" downloading programs like "Game of Thrones".

Clandestinely, and possibly unconstitutionally, attempted to shift the costs of Internet Surveillance, Collection and Aggregation onto Telecomms providers, and via them onto consumers, and without enforcing adequate security controls and provisions:

The Collection, Storage and Access of National Security Data is a wholly Government activity, that must be funded and controlled by them. It is not, and cannot be, a private sector responsibility.

Even a partial record of data requests made on the free-access (no warrant needed) datasets is a highly valuable and sensitive National Security Asset. i.e. If someone inside Telstra is paid by "black hats" for that information, it goes unrecorded and is a serious security breach, deeply and irrevocably affecting ASIO et al's operations.

The Access, Control and Curation of these datasets is a highly classified activity, to only be carried out by suitably cleared personnel in adequately secure premises and under strict controls, logging and compliance with all laws, including Privacy.
Rather strict privacy, governance, compliance and monitoring/audit requirements apply to the storage, use, correction and destruction of classified datasets, especially collected en masse, about Australian Citizens & residents domestically.

Very different rules apply for internal/domestic and international Security operations. ASIS and ASIO are independent and separate agencies for very strong reasons.

Signals Intelligence, SIGINT, is now a critical and fundamental activity of National Security and Espionage activities. Without it, Britain probably would've been unable to survive the North Atlantic blockade by Nazi submarines, changing the outcome of WW II. It's subtle and has an extremely large "force multiplication" factor. Most people don't appreciate it's value, importance and complexity.

At the end of 2004, a decade ago now, the "Hackers turned Pro": the top computer hackers started selling their skills and services to anyone who'd pay. They stopped hacking for fun, and moved to making serious money. All those very bright, extremely capable PhD's in Maths, Physics and Computing from the Eastern Bloc countries that were suddenly unemployed or not being paid after the collapse of the Soviet Union: What were their options? Many got recruited into hacker rings: they could sell their wares by actively recruiting the Best and Brightest. It's not just to stay ahead of the other side, it's to maintain a constant stream of products for sale and to enhance their product lines and maintain their longevity. It's good business, like paying your key employees well.

There are two outcomes of this:

whatever I write publicly is giving nothing away to "the bad guys". They are well ahead of me in their deductions, capabilities and active probing of security and perimeter defences of every worthwhile target on the internet. This is a professional, for-profit, industrial-scale undertaking. The "APT 1" revelations of Mandiant/FireEye didn't tell us anything we didn't suspect, only confirmed it as fact. Like the 9/11 plane strikes: after the fact nobody could deny the threat is real.

The way the business of illegal hacking works these days, is just like in the legal business world: there are a very few core experts in a field that pass on, or sell, their services, knowledge and skills to everybody else. Whatever I write here is already common knowledge within Terrorist groups operating in, or considering, Australia, because they've been told by their experts. The bad guys are way ahead in this game. We have to catch up, and the fools trying to big note themselves in Parliament need to learn this and act accordingly.

The two big reveals by Brandis may sound simple, but I'd be mortified at their disclosure if I worked within the Australian Intelligence community. Giving away what you cannot do is a bigger misstep in Intelligence/Espionage work than anything else. These are not mere suppositions or possibilities now, following Brandis' incompetent and incomprehensible bumblings over the last week, they are now confirmed fact. They've become so because the plethora of press statements and appearances has revealed and confirmed capability gaps and details on Intelligence operational matters.

Publicly stating that ASIO and the AFP need a data retention period of two years exposes critical operational details (time limits) of the Social Network Mapping done by them. The 1987 Fitzgerald Royal Commission in QLD did not use relationship mapping, while they were central to the 1995 Wood Royal Commission in NSW. Nearly two decades on, just how much better are the systems that ASIO and the AFP routinely use? Massively so.

Retro-actively mapping who knew whom, and inferring relationships by time-correlating locations, activities and accesses is an important operational capability for Law Enforcement. Once a person is fully recruited into serious crime or a terrorist cell, they "go black" and are untraceable. Being able to retro-actively trace them before they took measures to operate anonymously is a critical function in mapping social networks.
Brandis just ham-fistedly announced to every criminal and terrorist organisation on the planet that we can only look back two years. That gives them a firm "window" from when they go invisible to when they can safely start operating. The whole point of small, independent cells is to limit the damage done when a single cell is compromised. Who knew he could be so stupid?

The Domestic Intelligence Agencies need to map Social Networks of potentially problematic individuals retro-actively, because once they're noticed, they'll be taking counter-surveillance measures and invisible on-line. The two years of metadata retention is to identify potential recruits, and the people in their circles at the time. Given identified targets, modern physical surveillance can be ordered to rule people in, or out, based on suspicious behaviour and activity. Using untraceable internet connections might be classified 'suspicious' by ASIO, even if innocent.

The other major reveal is a capability gap that brought the heads of ASIO and AFP security out together in public. Brandis was very clear he needed IP numbers, not URL's and text addresses.

The only IP number that isn't clear these days is the Source. By definition, Destination IP addresses are public on the wider, public Internet.
There's a major intelligence gap, a massive blind-spot, for ASIO and the AFP:

They cannot traceback the source of mobile devices.

Geoff Houston, one of two people to run AARNET in its first years, has written extensively on the problems of CGNAT (and this piece) - Carrier Grade NAT (Network Address Translation), necessary to handle mobile data on 3G/4G networks.
The UK riots were only able to occur because of similar 'intelligence gap', or blind-spot, of the British Police: they were unable to intercept, or issue warrants to intercept, messages on Blackberry phones. Once the darling of Corporate Warriors, they'd become the poor man's smartpone - but still maintained secure messaging uncrackable by Police, performed in Canada, outside the jurisdiction of the Brits.

As for the "backdoor" implementation of draconian "illegal downloading" monitoring and enforcement, that were knocked back in February, Greg Jericho reported this:

The terror justification lasted less than a day because on Wednesday the Prime Minister made it clear the new laws will be used for “general crimes”.

The tool used to download videos like "Game of Thrones" is, and was named back in February, by Brandis is Bit-Torrent ("Torrents"). It doesn't use URL's because it's not a web browser. The people Brandis would like to categorise as "Internet Pirates" are solely identified by their IP numbers. Who they connect to, coupled with the size and duration of the download, will uniquely identify what they downloaded. If "torrent freak" can track and report what's downloaded, so can the AFP.

The two-year limit means if you're identified as a potential "Pirate", they can drive back in time and hit you with multiple charges. You go from a minor infraction to being a major criminal with an overwhelming count of charges. This is the same strategy executed in the USA: identify and persecute some harmless and poorly resourced individual. They get "prosecuted to the full extent of the law" and hit with punitive fines that total in the millions. This approach to "make an example of them" is very poor policy and can only backfire on this government.

We can infer that Snowden didn't tell us about the most important, most sensitive capabilities of the NSA and the other secret Agencies employing over 1M US nationals. Intelligence Agencies, since the 1940's and Colossus at Bletchley Park, have built or bought the largest, fastest computers available. Whatever Google can do, you have to assume the NSA and Cyber Command can do as well or better. Both groups purposely hire the Best and Brightest. They entice & engage them because of the size and scale of the problems they work on: they'll be right at the leading edge, solving hard problems for the first time, "going boldly forth...". That's catnip for geeks right there.

If Google can analyse and categorise images and videos in real-time, then so can the NSA and friends. The best high-performance, real-time computing people come out of particle physics, like CERN's Higgs Boson facility. I'd expect both Google and the NSA recruited people from those teams.

We know Google and many other Internet firms, even Apple, can automatically recognise and match faces. Google can automatically recognise images of places, buildings and scenery as well. But over the last 35 years of the PC revolution, we've seen a complete transformation of the field of advanced image processing:

The Boston Marathon bombing confirmed that Law Enforcement agencies have the capability to stitch together, in space, orientation and time, many disparate images and videos. Thousands of people uploaded their smartphone videos of the day, to create the largest publicly known multi-viewpoint CCTV surveillance system.

given multiple viewpoints, accurate 3-D models can be computed, it's the basis of CT scans and stereogrammetry, used by map makers to transform photo from planes into accurate terrain maps.
Given a 3-D model of a scene, it can be rendered from any fixed or moving viewpoint, that's the basis of CGI (computer generated images) used extensively in movies etc. You see this every time you use Google Earth in 3-D mode.
Given a 3-D model of a scene, even patchy, time can be added as moving objects are tracked. There is already an operational "synthetic CCTV" system that joins the video of the streets taken from on-board cameras of buses. Think what Google Street View might do if they sent cars along every 5 minutes.
Astronomers have worked on Digital Imaging and improved it out of sight in the last 3 decades. They 'stack' low-res images into high-res, high-dynamic range. That's why Hubble is no longer the best telescope on, or near, the planet. Given 10 or more low-def images of a scene from smartphones, you can have a good high-def image.
Multiple images from digital cameras can be merged to produce better than 1-pixel resolution. The Maths isn't difficult to infer edges and movement when there's only partial information. That's why we can digitally restore old, degraded movie films. There are many images of the background as a scene is panned. They can be combined into a high-resolution, very high-fidelity composite.

Google (Picasa & Youtube), Facebook, Twitter, Instagram etc have massive collections of public images. I'd presume that these "Open" data sources do not need warrants or even special arrangement for any Intelligence agency to download every image and frame of video.

While the UK might have the largest collection of fixed CCTV cameras on the planet, the USA, via its plethora of smartphones, probably has the highest indoor and outdoor coverage of synthetic CCTV cameras: the aggregation and combination of all public smartphone videos on the web. There are now no significant events that aren't recorded and uploaded, usually by multiple smartphones allowing image enhancement and selected viewpoint generation through time. Even in near real-time as feeds are uploaded.

If all this can be done for video, what's possible with the included audio? Especially as we now have very good public voice recognition systems. Think Apple's Siri.

Snowden's revelations, that I've read, don't mention what the NSA can do with smartphone videos and images. Given the availability of data, ubiquitous coverage of smartphones and public capabilities of Google et al, I find that a surprising omission. It's also consistent with a concerned whistleblower who's intensely loyal and consciously acted to not harm his country's interests to suppress that information.

These are suppositions or inferences on my part, not facts. We might guess these capabilities are routinely in use, and infer how much more could you do with accurate identity databases, but as yet, we don't know. We saw impressive "fly throughs" in the 1993 film, Jurassic Park (by SGI computers). Twenty years on, how much better is possibly on a $5,000 work-station? A million times or more.

So, is Brandis a mere bumbling, stumbling fool with no technical comprehension or something much worse? Only history will inform that question.

Malcolm Turnbull also confirmed Source IP numbers were 'invisible' to Intelligence Agencies:

What the security agencies are after is that (retention entities) should be required to keep that record for two years. And the purpose of that is: if at some point, it is noted that a particular IP address, which is one of the bundle of IP addresses that belong to Telstra, has been on a criminal web site and engaging bad people - you know, whatever - then the police can go to Telstra and ask ‘IP address 12345, who was the customer that was using that on the 4th of June 2014?’