What does security mean #3: What does security mean for your Linux kernel?
by Roland Gharfine
“I have a plan of action, but the game is a game of adjustments." - Mike Krzyzewski
Welcome to today's practice, I am your coach for the day, Roland Gharfine. I hope you've had a good meal, because we're about to run some hard drills and defensive schemes.
Before we get started on our strongly opinionated journey, I'd like to tell you a little bit about me and about what I do first.
It may surprise you that I am not in fact a Basketball coach. I'm a hybrid between a CISO and a security engineer, and I've had the good fortune of molding a company's security framework in an almost single-handed manner. In short, for the past 7 years, if it had to do with security, whether operations, strategy or policy, I had my hand in it. I have some other qualifications that may not be strictly relevant to the conversation, but feel free to check out my Linkedin profile if you're curious.
Now, let's talk defense. We will detail some different preventive controls for Linux kernels and make analogies with some basketball defense schemes. Although the context was different, I discussed the importance of preventive controls in this post. The fact of the matter is, as I mentioned there, all controls are not created equal. You can't really discuss complementary controls that really aim to compensate the preventive ones if you don't have the prevention solidly in place. In basketball terms, this would be like building a team with 8 role players and no starting point-guard. If you got that reference, you'll love this article, so read on, and let's turn our Linux kernel into a fortress.
We will generally follow a pattern in our sections: make the analogy to the defense scheme, explain what can happen and how the control helps us, and then provide some examples of tool support that would help us implement our defense in question.
Zone defense fundamentals: File system integrity
Catch-all solutions are helpful in prevention, and kernel hardening is no different. We shall call these strategies "zone defense", mainly for the two following reasons: they can help us solve a lot of problems, and they have a telling weakness. Although they are not enough on their own, they are still very useful and must be implemented nonetheless.
Chief among those zone defense fundamental controls is file system integrity. What is it? In essence it means implementing controls that will at least notify us when the main files of the OS are being modified. Ideally the control should do 3 things, and notifying us is only one part. The other two would be stopping the change, and sandboxing the application that was trying the change in the first place. We'll talk more about sandboxing later, but in essence, if a particular player is torching your zone defense (a process has some success bypassing your file system integrity controls), usually with jumpshots (bypassing the integrity checks), then catch-all solutions will stop working. We'll get to that, I promise.
What you need as a control is immutability, at least for the essential kernel files. There are factors that can play into whether you can implement immutability in the first place, but assuming you know how to make volatile immutable deployments or setups, and that your use case supports them, then it's definitely the way to go.
Controls for immutability include trusted OS loaders (like TrustedGRUB), checksums, a change management enforcement, audits, anti-rootkits, and antivirus software. A combination of those methods will allow you to enforce the immutability of your kernel from boot time to termination.
Rootkit software will generally differ from a virus in that it is not a single executable or process, but rather a patch to the kernel that will allow the attacker to abuse it, a change to its files that is passive in nature but can be used and reused. Antivirus software is clear enough, but the telling point here is that most system administrators, even experts among them, will shrug off the need for an antivirus on Linux. This is a highly misguided and quite frankly naive way of seeing things. Just because Linux viruses are less prevalent, doesn't mean they don't exist, and if other layers of defense fail, an antivirus is the only entity standing between a malicious process and your kernel modules.
Notable native software that can be used as an anti-rootkit, change management enforcer, and integrity checker, is auditd. Auditd comes bundled with most Linux distributions, and can be deployed as a service. Using various rules, you can prevent or at the very least register changes or even read access to crucial files.
An example of an antivirus software that supports Linux is ClamAV.
Finally, checksum validation can be easily done with hashing algorithms which have programs that come pre-installed with Linux (try sha512sum on a file and see what it can do).
Zone defense evolution: MAC, RBAC, and least privilege
Least privilege is one of those terms that seem nebulous and abstract, and that's because it is linked to abstract and nebulous process-related quirks. I dive more into least privilege and what it practically means in this post. For our purposes right now, we will just clarify how it can help us with our defensive scheme.
First of all, let's mention MAC (mandatory access control). Whenever we say MAC, we don't just mean permissions, but rather policies, hence the mandatory part. Essentially, it's another catch-all control, but an advanced one, in that it implements the preventive element by design.
A combination of RBAC and MAC can be really powerful: assign roles to any entity that may launch processes on the system, and enforce some restrictions on it using a policy based approach. It's a fantastic 1-2 punch.
The key here is to understand that processes are the actors that deal with the kernel, not humans. Access control is therefore not limited to individuals, but rather programmatic entities and the identities that they assume, for example, the way a service account can launch a process, which in turn can use a system call.
System calls are the big element here, as they are really the interface to the kernel. Little does the user or service account know that they are actually interfacing with a mirage: the shell. The shell has no power in itself without its own interface to the kernel, which are system calls. We'll later talk more about how to target individual system calls with our defensive schemes, but this is the key to most things related to the kernel.
Notable tool support for MAC is SElinux (security-enhanced Linux). SElinux will allow you to define entities called contexts and enforce policies on which users can use which files under which context. Since everything is a file on Linux (including network sockets and kernel modules), this is an approach that will encompass everything. Contexts will in turn define permissions, which is something we will see in sandboxing. Come to think of it, SElinux has a much wider reach than just MAC. It's an all-in-one tool that includes MAC.
More zone defense: namespaces and control groups
You may decide, most probably through risk assessment, that access control and integrity are simply not enough for your highly secure use case. What's next on our list of strategies?
Two things come to mind: namespaces and control groups (also knows as cgroups).
A namespace is an isolation technique that the Linux kernel can use to prevent processes from interfering with each other. There are several namespaces implemented by default on Linux, namely: mount, UTS, IPC, and network. Namespace isolation is particularly useful in a containerized setting, because it prevents the container from seeing all the processes running on the host kernel.
A cgroup is a collection of processes grouped by similar criteria. It's a more flexible example of the "nice" prefix command, which provides some security features. There really isn't much to explain about cgroups that doesn't qualify as technical minutia, but rather you just need to know that you have tools at your disposal that can help you isolate processes (the kernel only knows of processes, remember), which is a jumping point to tailoring your defensive approach, even if isolation is a catch-all solution in itself.
Box and one: Limiting capabilities
You're watching there from the sidelines, as one player keeps hitting your zone defense with 3-point shots. Your defense is set, but that opponent player just has a hot hand tonight. You call a timeout, fuming and yelling profanities, but you soon realize: your players aren't doing much wrong. It's just that your zone defense has limitations, and that player is exploiting them. What do we do next? Switch up our scheme with a complementary twist.
We're getting into more advanced defenses here, deviating from the catch-all approach. A box and one defense is a zone and man-to-man hybrid: The zone principles still apply, and therefore we still have our catch all solutions, but that one player will receive special attention, and will be faced with a man-to-man defense.
What does this mean for our kernel hardening? Well, we can limit some processes to certain capabilities rather than just trying to catch their malicious activities when they potentially occur. Capabilities (also known sometimes as POSIX capabilities) are general categories under which each type of access to the kernel modules will fall. For example, setting the date will need the SYS_TIME capability, and modifying networking options will need the NET_ADMIN capability. Again, these are not permissions for a user, but rather capabilities that can be used by a process, just as a reminder to move away from traditional access control ideas and into MAC concepts.
Limiting capabilities for certain processes will allow us to keep our catch-all approach, as capabilities give us access to system calls in bulk, but will allow us to tailor our approach to some processes in order to protect our precious kernel from any potential misuse.
Man-to-man defense: Sandboxing to limit system calls
So far we've discussed solutions and preventive controls that will target many or all processes at once. However, sometimes this is not enough, as this one opposition player can be so malicious to our defenses that they require individual attention. Man-to-man defense or even double-teaming that player requires a feature-rich and customization friendly preventive control. Enter sandboxing, the concept of showing the process a version of the kernel that is not quite fully-featured, and therefore preventing the process from escaping our sandbox and enforcing restricted access.
This is where the anlogy slightly breaks down. Implementing our man-to-man defense to target specific processes and system calls does not mean we will stop implementing our zone defense and deviating from our catch-all controls. Rather, we will implement those controls in conjunction. This is more akin to the way a basketball defense will act over the whole game: adjustments, and combinations of many defensive schemes in order to achieve success.
What can sandboxing allow us to do? Well, many things, we can target individual process and prevent them from running certain system calls, or even accessing services and external resources. We can, for example, prevent a process from using network sockets, or binding to an address. We could prevent a process from accessing files, running with certain parameters, or writing to certain memory address ranges. The list is endless, really, and the idea is to limit a process to the uses it was intended for. This way, even if the process is hijacked by a malicious actor (a common approach that many exploits use), it won't be able to cause as much as harm as it otherwise would have.
A two-item list is all you need to remember in terms of tool support: seccomp and apparmor. Yes, many other tools exist, but this combination will probably cover the vast majority of possible use cases. App armor can learn from your past syslogs and discover what resources and system calls are needed by each individual process, and then ask you to confirm that this is indeed normal usage. Everything else will, of course, be blocked. Seccomp will work on a policy-based model, and allow you to write rules that will dictate how processes can run.
Communication on defense: Intrusion detection/prevention
Any basketball or sports coach in general will tell you: defense is 50% communication. Without some overarching visibility that can guide the various pawns and tell them where they should pay the most attention, it's all glorified chaos.
Once more, a useful analogy, and we will absolutely capitalize on it. What is the equivalent for kernel and system security? IDS/IPS systems. Specifically, we are discussing host-based intrusion prevention, because network-based solutions exist as well, and although the techniques might be similar, the end goal can differ.
The general consensus splits those intrusion prevention systems into two categories: signature-based and anomaly-based. Both will generally make use of machine learning, and both present advantages and disadvantages. While signature-based systems come pre-trained and ready to go (since they just compare events to a list of signatures and attempt to find a match), they are limited in their ability to detect malicious events that are out of their pre-trained realm. In addition, they will also need constant updates to their signature database. On the other hand, anomaly-based systems can detect malicious events that are unfamiliar or unknown (zero-day attacks), but do require a period of training (usually called learning mode) in order to discover what normal usage looks like. They will also require substantial tuning in order to reduce false positives.
After that long-winded explanation, why did we liken intrusion detection to communication in a basketball defense? Because intrusion detection is what provides us with the visibility and information about where our defense might be going awry. Without an IDS you will simply keep making the same mistakes again. Intrusion prevention is your feedback loop that will close and stabilize your system (hello mechanical engineers... wait, why are you reading about kernel hardening?).
An extremely popular intrusion detection software is Splunk. Although (at least in my opinion) there's a substantial learning curve involved in it, it's still a formidable tool that can supercharge your kernel defense. Other notable tools include OSsec and Samhain.
Some advanced topics
No basketball team implements one defense for the whole game. In fact, combination defenses and hybrid approaches are really popular in basketball, and as I always insist, they are popular and useful in cybersecurity as well. As a result, no one defense will be enough, and your team must know all the schemes and techniques in order to mount a successful defense.
We must also note at this point that we are barely skimming the surface on many of these defensive controls, and even though the topic itself might be advanced, we have just begun the discussion. Kernels are beautiful software entities in that they make an extremely complex thing like machine interaction extremely simple, and so they must be complex in nature. Kernels are really both simple and complex at the same time: complex to understand, modify, or build, but very easy to use.
As usual, I'm inviting you to strike a balance. How? Well, by understanding the advanced, using the easy, and securing in a simple and effective way.
Some advanced topics that we haven't really covered include (but are not limited): address space protection, buffer-overflow prevention, trusted path execution, init-kernel mutual authentication, and randomized PIDs. Do your research, and come see me after practice one day, I just might have some interesting information for you.
We're talking about practice, not a game, but practice
Thank you for all your hard work in today's session, and I do sincerely hope that you are benefiting from my posts, and most of all, enjoying them. In the words of the legendary Allen Iverson, this is just practice, and what really counts is how well you perform in a real competitive game. I will also spin the retired player's quote into this advice: go out there and get your hands dirty. Research the terms you have learned today, fire up a Linux machine, and try some commands to see what effect they have. There is no substitute for real-world experience, so go out there and make your own luck by getting it. Practice, practice, practice.
Please do not hesitate to visit your coach after practice, and contact me with suggestions, criticism, rebuttals, or even questions if you think you or your organization could use my experience and opinions. Just saying hi is once more (and forever will be) perfectly fine as well. Also, find my posts on the hashtag #askroland, and follow it so you make sure that you always see me in your feed. I just enjoy smart discussions, and I hope you and I can have them.
Hit the showers, practice is over.
The article originally published at: https://www.linkedin.com/pulse/what-does-security-mean-3-your-linux-kernel-roland-gharfine/?trackingId=gu1Bhu6LTGiPUZtVrNPN1A%3D%3D