Add data - Splunk Tutorial

- [Instructor] In order to give us some realistic logs to work with, there's a Python script in the exercise files called Log Generator. To keep things simple, it runs inside a Docker container. So just extract the exercise files and be sure you're in that log_generator directory. Here I've just copied it into my home directory. Now the first thing we'll need to do is build the container image. So that's just docker build -t and then log_generator. And then we'll need a dot. That's to specify the local directory. And that should just take a second, especially if you already have the base image downloaded. Then to run our container, we'll just use docker run -v, and that's the command to attach a local directory inside the container. Now the local directory we're using is ${PWD}, and that's just a way of saying the current directory. And then :/app. So we're going to install this, attach this directory into the app directory on the container. And then we'll say -h for the host name, and we'll name it web. And then -d, that's to tell it to run in the background and keep running. And then we want the image name we just created, which is log_generator. And hit Enter. And you should see a long string of characters like this if it works successfully. Let me just clear my screen. And before we look in the web browser, let's just do an ls. These are all the files within this log generator, and there's some things we'll use here later. This docker-compose we're going to use a little later in the course for some network-based testing. And if we look in the out directory, we can see there's two log files being generated. Let me just cat one of those. So this is the access log. Looks a little strange because I'm kind of zoomed in here, but this is like a web server access log. And if we look at the secure log, this is like a Linux security login attempts and things like that. So let's clear the screen. And we can open a web browser. And I'll just enter localhost:8000. So that's our container running our Splunk server. We'll type in admin for the username and then whatever password you set up. And I'll just say Got it on this popup. Now what we're doing for this exercise is just kind of simulating monitoring a local file that's actually on the server itself. That's why we did that -v in Docker to attach that directory from the Docker container and from the Splunk server. So what we want to do is go down to Add data. It's kind of in the middle toward the bottom here. And you can take this tour if you like, it's actually helpful, these little UI tours. But I'm going to skip it for now. And there's some options here for it says the more popular data sources, but we're actually going to use just old fashioned method here of directly adding the file. So it's Monitor down in the center. And it says files and ports on this Splunk platform instance. Since we're mapping that directory into the Splunk server, it sort of thinks that the file is on this instance. And this is just so we have example data to look at, this isn't really a process you'd use in real life. So you can see here we've got a few options, but we just want to go up to the top one, Files & Directories. And there's some description. Splunk is really good at providing like inline documentation and links to the things you might need to know. There's a useful context here. If we were running on Windows, the path here would look a little different, and it's kind of pointing out that they're back slashes instead of forward slashes. But we'll just go Browse. And since this is a Linux-based container, we have the Linux directory structure here. Here's that sample_logs directory that we attached in to the container. And because our log generator is running and it's sharing to that same location, it's actually put those log files right here onto our instance. So let's do the access log, and we'll walk through this one. And then after we finish this, you can just before you go to the next video, go through the secure log following the same steps. So click Select and be sure it's set to Continuously Monitor, because the log generator is going to be generating data all the time. That's kind of why we're using this format instead of a static file. It'll be a little bit more like monitoring live data. So we'll click Next. And here we can see some of our sample data is already coming in. And this looks like web server access log data. You can see the URLs and what kind of web browser. There's a date here, things like that. And on the left there's a dropdown box that says Source type: access_combined. And if you open that up, you can see there's a lot of options here of kind of built in source types. So for example, under Database, mysqld, if we were monitoring a MySQL database, we'd have the right format for that. There's quite a few here. And I'd say most pretty common log formats will come up here. This access log is a pretty standard type of format, so it's not a surprise that it's included. And as you run through this on your own for the Linux log, you'd go down to Operating System, and linux_secure is the log format to use for that one. But for this one, we'll just leave the default. And quite often Splunk will detect the right format as it did in this case. So we'll click Next. And there's some things about input settings here that we don't need to worry about, although I like to change this host field value just because this is just a random string generated by Docker. So let's just change that to web. And we don't need to worry about the index for now. So we'll click Review. Just take a quick look at this and make sure that matches what we talked about. And then click Submit. And that's it, our data's actually being actively ingested now. So there's a few options here, but the one we want is to just start searching. And I'd encourage you to pause the video and take this tour if you're following along in real time. But I'm going to skip it. And they have another popup here, this time telling us about the dark mode, which I kind of like the look of, but we'll go through this course in light mode. We'll click Skip tour. And here you can see our logs are already available in our search interface. We'll go much deeper into how to use this search application in later videos. But if you want to spend a little time right now just poking around and trying things out, these dropdowns open up and you can see a little more context. There's information up here that can be edited and searches that you can do and things like that. So spend some time poking around and then go through and add the secure log and then you'll be ready to move on to the next video.