Critical strike: China's hacking training grounds (PART 2)
A deep dive into the technical elements of the "Expedition Cloud" Cyber Range designed for the Chinese Ministry of Public Security.
Welcome to PART 2 of our exploration of Cyberpeace's “Expedition Cloud". Today we will do a more technically focused analysis of the core elements. If you haven't read PART 1, I would highly recommend to do so:
Right from the start, let me tell you, we won't be able to show WHAT kind of operations they conduct against targets. We don't have access to the "weapon” data.
We will base our analysis mainly on an “installation manual”, UML charts and some source code. The manual focuses mainly on setting up the different network entities, communication tunnels, subnet configurations, Redis database and respective Openstack services etc. which we won't cover to much in detail.
But in combination with the other data, we can infer actually quite a lot on how the system operates.
We will chow through it bit by bit and I assume that everyone who has been involved in building more sophisticated cyber ranges will see many familiar elements along the way.
Refresher
Lets recap what we have so far in case you can't be bothered reading PART 1.
A Cyber Range setup (C-R), commissioned by the Ministry of Public Security of China, to train government cyber operatives in attacking other nations critical, digital infrastructure.
The C-R is consisting of four main elements. One is the Intranet (I-N), the second is the external network (E-N) with its worker nodes (W-N) and the Relay (R) and the "targets”, simulated infrastructure hosts for operatives to run missions against.
As a refresher here is the original layout:
This C-R is not just existing in a lab setup but stretches into the open internet based on what we found.
Data exchange between the W-N and the I-N is done indirectly and passively via the Relay (R). Neither the “Worker Nodes” nor elements of the Intranet send data directly to each other. The R is a “dead-drop” where data is posted and picked up by the operative elements. Data from successful missions are send first to the W-N and get then send to the Relay for ingestion into the I-N.
Each of the components do possess very heavily segmented networks on their own. The I-N and W-N each hold several NICs that separate data processing, storage and access, while trying to hide it all behind one public IP address. The I-N even possess a segmented one way data flow gate, to prevent easy data exfiltration into the open internet.
Dashboards
Before we go into talking about all the aspects in detail, let us take a look at some dashboards of the project.
We will draw many insights from them while showing more screenshots along the way, but it is helpful to know of their existence.
Generally they are the back-end that administrators, supervisors and team members can use to keep an eye on the overall operations and tasks, configure scenarios, create new users and assign missions specific tools. Different versions exist for different roles inside the Cyber Range.
Here are a few examples to illustrate the dashboard for the admin side of the C-R ( more details on certain aspects later ).
There is also a second dashboard, operated ON the W-N ( not the I-N ) that enables additional access to the hosts and their operations, like setting up VM's, manage user access, roles etc. You can see a slightly changed UI design. Despite this vue UI is operated on the node that is accessible via the public internet in theory, it is not easily uncovered.
This W-N dashboard is interesting as it has a decoy function. The real vue dashboard login is only revealed if a valid “authentication token” is presented when accessing the public front end. Otherwise, a decoy webpage is presented. Something that might resemble this technique !? In the project files we have they show a generic website like this:
We will come back to it a little bit later.
Targets
What do we know about the “targets” ? Based on the installation manual we know that there are internet based “nodes” that are linked to objectives. Most likely meant to be the simulated target infrastructure ( though by all means, it could be real targets too, but the earlier document has no reference to it, only training ).
As the installation manual only focuses on setting up the I-N and E-N parts of the C-R, there is no explanation of how a target infrastructure is setup. Our interpretation is that they run on separate instances ( most likely VPS ) outside the I-N/E-N structure, as illustrated in that part of the layout:
What happens on the Worker-Nodes (W-N) ?
As stated before, the W-N is where all the heavy lifting is done. Comprised of two elements, the compute node and the control node, this entities located outside the internal network is the main operational element of the Cyber Range.
The W-Ns do exist outside the internal network and have their own public IP address, but are all integrated into the REDNET to be accessible directly from the I-N. An internal network-style mapping that still creates a cohesive visibility of the components for the internal network, without being “physically” connected, implemented via OpenStack.
That means, from the internal network, all the W-N nodes are accessible via an internal IP address, not a public one. The data routing seems to happen though via JSON files stored and picked up from the Relay server. So, instead of having a direct routing between connected networks, "messages” are sent to the Relay so the W-N and the I-N can communicate with each other. The Relay is actually pretty much the only element of the C-R that the W-N interacts with on a regular basis.
An Open Stack powered structure, run on top of CentOS, provides the framework for the Virtual Machines the operators connect to, to conduct the "tasks” and “missions". The VM's will be installed beforehand by copying OS installation packages directly to the host via ssh connections. It is a two stage process: 1. copying the VM installation package to the W-N directly 2. register the VM with the wider system.
Via the built in dashboard, the VM will be hooked into the logic of the Cyber Range. This way the I-N can work with it. Remember, the W-N and the I-N are not directly linked with each other, but stay connected via the Relay. So the I-N needs to have some slim “mirror” running on its side that is filled with the real world data provided by the VM on the W-N. This logical link needs to be established inside the admin system.
As mentioned earlier, this W-N admin dashboard is hidden behind a decoy website. Code fragments show that it is only revealed if an valid authentication token is presented, but how this token is created we don't exactly know. We found references to a USB-Key, to an QR code driven app authentication and of course normal login as well ( most likely a combination of those ). This decoy function can be switched on and off by an administrator.
The W-N can stage several VM's, running Kali, Windows, MacOS, Solaris Linux etc. plus it can also provide Android phone environments. It doubles as master control node with admin functionality, that controls and supervises the Virtual Machines conducting tasks according to a scenario etc.
When running operations, the VM's network activity will be recorded, including streaming video from the VM's desktop. All this data will be stored on the W-N, processed and then passed on to the Relay server. From here it will be picked up by the I-N for further analysis, after-action, evaluation and mission specific status update.
There seem two main ways for administrators or supervisors to control the W-Ns. One is via the above mentioned dedicated vue dashboard ( that only reveals itself under certain circumstances ) or by using a dedicated phone app.
This phone app accesses the W-N and a special subnet via an dedicated VPN service and then uses a REST API to communicate with the backend. Interesting enough we found an patent filed by Saining Technology referencing a potential VPN connection method to facilitate this connection ( it seems to be OpenVPN based ).
We could not get a complete version of the phone app, but got a mock up design of its messenger system and rough layout ( "cloud notes”).
Each W-N seems to hold its own version of scenario data, permission sets, user permissions and roles, "weapon” setups and so on. It gives the impression that there can be quite a number of W-Ns, all with their separate focus. The system seems to refer to it as "CPCloud” (CyberPeace Cloud?).
Extracting referencing paths from a vue backend we could reproduce the overall structure of the W-N based dashboard controls ( though not the direct UI, as this elements were missing from the vue framework ):
This seem to tie into a REST API that serves the vue dashboard as well as the phone app as mentioned earlier. As you can see, the options are quite self explanatory and involve mainly the administrative back end operations of the W-N. Many of those functions will be replicated by the dashboard of the I-N we will show you a little bit later.
“Stealth” Links
The second most interesting element are the 迷踪 ( MiZong ), or stealth links. First mentioned in the Installation Manual's introduction page, it says: 目前系统支持的隐蔽链路有迷踪设备；
They are most likely proxy nodes on separate VPS used to hide the operators paths to “targets”. We don't know that much about them, except that they appear in several panels of the I-N dashboard ( including a world map showing all the possible proxies) . According to the dashboard logic we see they can be assigned to specific operators or scenarios it seems.
This so called “迷踪设备“ could be implemented via some V2Ray/Shadowsocks configuration as the dataset had some “ShadowsocksX NG_diagnose_20201119_121940.txt” files included, but this is just a guess. This file is just a client side diagnose which also could come from the developers machine itself and may not be connected to the project.
The Relay
We mentioned the Relay server before but lets take a closer look. Located between the I-N and the W-Ns, it is the mediator between those two separated networks and the “link” to transfer data. All of it is done in a rather passive way, as the Relay in itself is not talking directly to either component, but provides a means to securely store message and commands ( in form of JSON files ), mission data and video recordings.
The W-Ns upload and download data via SFTP. While the I-N passes or retrieves data via its “optical gateways” that in turn then use SFTP to convey data to the Relay, each on its separate network configuration. In this function, the Relay also prevents an elements of the I-N to have to face the public internet. Unfortunately not more info is provided in the documents we got acquire.
What happens on the Internal-Network (I-N) ?
Now let's take a look at what the administrators, supervisors and team members can do on the internal network. The main access point for most users, the data will be eventually stored here, processed and evaluated. This is also where the “main control room” is located. Thanks to the installation manual and many screenshot of the admin dashboard, we can tell a lot of the capabilities of the system.
First off the I-N holds all databases with the core data. User accounts and permissions, scenario templates ( as mentioned in PART 1 ), authorizations, logs and task results etc.
But then there is also the capability to control the Cyber Range.
We have screenshots here of how missions are assigned. An supervisor has to start a mission, someone higher up the chain needs to authorize it and finally operators are getting assigned to it.
The supervisors can also see all active VM's in operation. They can taken them down as well.
Same applies for Android VMs it seems. You can see the Rednet IP addresses clearly. This is how the Rednet gives the supervisors and administrators the impression, that all "devices” are on the same network.
After an operation is completed, the supervisor can instruct the system to download all the raw data from a particular VM. We can also see that there are options to inspect “weapons” and "combatants”, though we don't have more information on how that looks like.
A world map presents an overview of ongoing missions and tasks, evaluation of all the operators, system overview and potential "link nodes” to give operators the potential to hide/reroute their attacks via a proxy ( see above ).
For replay, evaluation and after-action analysis, the supervisors can request the log files of the VMs from the W-N. The logging of the network activities seems to be done by a "Wahzu” based service on the W-N.
Supervisors and team members also have a way to communicate with each other via an internal email style communication platform. In the documents we have seen it is called "809 Intranet Platform" ( a temporary name most likely ). There seems to be also access to some "knowledge base” articles on the left as the Cyber Range also provides some repository of past actions, tips-and-tricks and tutorials: 密码破解能力, 网络调查探测能力 or 品隐蔽能力.
Final thoughts
I guess for seasoned network administrators and cyber range designers, this whole setup might not be massively surprising. Spreading it across the public internet, gives a few advantages compared to have a lab-only construct.
As stated above the dataset we obtained unfortunately does not include the "weapons” (payloads, exploits etc. ) they use in this cyber range to conduct their missions, nor do they specify concrete mission/task details. That would have been of course the most valuable information to most members of the cybersecurity community, but as this elements are most likely specified by the authorities and not Cyberpeace, it would have been surprising to find it in there.
What is clear is that the design is pretty sophisticated and the builders have put in some thought of how to keep the whole Cyber Range hidden away from prying eyes.
Interesting is that this Cyber Range was commissioned by the Ministry of Public Security, not the Ministry of State Security. Later is usually tasked with dealing with “outer national” affairs and espionage. As this Cyber Range was designed to train hackers to find weak spots and exploitable aspects of foreign digital infrastructure, this seems an interesting choice. Not that much is known of how the cyber operations landscape is structured in China, it still strikes us as noteworthy.
Apart from building capacities through complex training scenarios, the Cyber Range also provides an extensive logging mechanism of all the data produced during an “operation". Robert Belch of XSOC Corp sees a lot of potential in using this extensive datasets to train AI systems down the road:
“Data from cyber-range exercises, especially replay attacks, telemetry captures, and blue-team vs. red-team simulations, can absolutely feed AIDA (AI-driven Data Attacks) modelling. These datasets have high utility because they contain both the signal dynamics and the defensive response patterns that AI systems can learn to exploit.”
We have no direct mention of this intention in the dataset we have but as we notice a rising utilization of AI in the Chinese cyber security scene, this conclusion doesn't seem too far off.
Now that said, we all know that it is not only the Chinese that run such "close to reality” training facilities. By now it is pretty standard with most countries that participate in cyber warfare. The only difference there is, China is able to scale into spheres most other nations can only dream of. Then there is also the suspicion that such a setup can be used for actual offensive attacks as well, though we have seen no evidence wit this particular Cyber Range.
If you have more information on this story either write to us here or use our dead drop under deaddrop.netaskari.online
Man, it really sounds like Information warfare is continuous preparation, not just wartime activity. And centralized command with distributed execution is rather... effective.