How-To Tutorials

In this article by Maya Posch, the author of the book Mastering C++ Multithreading, we will learn to work through and understand a basic multithreaded C++ application. While generally threads are used to work on a task more or less independently from other threads, there are many occasions where one would want to pass data between threads, or even control other threads, such as from a central task scheduler thread. This article looks at how such tasks are accomplished. Topics covered in this article include: Use of mutexes, locks and similar synchronization structures. The use of condition variables and signals to control threads. Safely passing and sharing data between threads. (For more resources related to this topic, see here.) Safety first The central problem with concurrency is that of ensuring safe access to shared resources, including when communicating between threads. There is also the issue of threads being able to communicate and synchronize themselves. What makes multithreaded programming such a challenge is to be able to keep track of each interaction between threads, ensure that each and every form of access is secured while not falling into the traps of deadlocks and data races. In this article we will look at a fairly complex example involving a task scheduler. This is a form of high-concurrency, high-throughput situation where many different requirements come together, with many potential traps, as we will see in a moment. The scheduler A good example of multithreading with a significant amount of synchronization and communication between threads is the scheduling of tasks. Hereby the goal is to accept incoming tasks and assign them to work threads as quickly as possible. Hereby a number of different approaches are possible. Often one has worker threads running in an active loop, constantly polling a central queue for new tasks. Disadvantages of this approach include the wasting of processor cycles on said polling and the congestion which forms at the synchronization mechanism used, generally a mutex.Furthermore, this active polling approach scales very poorly when the number of worker threads increase. Ideally, each worker thread would idly wait until it is needed again. To accomplish this, we have to approach the problem from the other side: not from the perspective of the worker threads, but of that of the queue. Much like the scheduler of of an operating system, it is the scheduler which is aware of both tasks which require processing, as well as available worker threads. In this approach, a central scheduler instance would accept new tasks and actively assign them to worker threads. Said scheduler instance may also manage these worker threads, such as their number and priority depending on the number of incoming tasks and the type of task or other properties. High-level view At its core, our scheduleror dispatcher is quite simple, functioning like a queue with all of the scheduling logic built into it: As one can see from the high-level view, there really isn't much to it. As we'll see in a moment, the actual implementation does however have a number of complications. Implementation As is usual, we start off with the main function, contained in main.cpp: #include "dispatcher.h" #include "request.h" #include <iostream> #include <string> #include <csignal> #include <thread> #include <chrono> using namespace std; sig_atomic_t signal_caught = 0; mutex logMutex; The custom headers we include are those for our dispatcher implementation, as well as the Request class we'll be using. Globally we define an atomic variable to be used with the signal handler, as well as a mutex which will synchronize the output (on the standard output) from our logging method. void sigint_handler(int sig) { signal_caught = 1; } Our signal handler function (for SIGINT signals) simply sets the global atomic variable we defined earlier. void logFnc(string text) { logMutex.lock(); cout << text <<"n"; logMutex.unlock(); } In our logging function we use the global mutex to ensure writing to the standard output is synchronized. int main() { signal(SIGINT, &sigint_handler); Dispatcher::init(10); In the main function we install the signal handler for SIGINT to allow us to interrupt the execution of the application. We also call the static init() function on the Dispatcher class to initialize it. cout <<"Initialised.n"; int cycles = 0; Request* rq = 0; while (!signal_caught && cycles < 50) { rq = new Request(); rq->setValue(cycles); rq->setOutput(&logFnc); Dispatcher::addRequest(rq); cycles++; } Next we set up the loop in which we will create new requests. In each cycle we create a new Request instance and use its setValue() function to set an integer value (current cycle number). We also set our logging function on the request instance before adding this new request to the Dispatcher using its static addRequest() function. This loop will continue until the maximum number of cycles have been reached, or SIGINT has been signaled, using Ctrl+C or similar. his_thread::sleep_for(chrono::seconds(5)); Dispatcher::stop(); cout <<"Clean-up done.n"; return 0; } Finally we wait for five seconds, using the thread's sleep_for() function and the chrono::seconds() function from the chrono STL header. We also call the stop() function on the Dispatcher before returning. Request class A request for the Dispatcher always derives from the pure virtual AbstractRequest class: #pragma once #ifndef ABSTRACT_REQUEST_H #define ABSTRACT_REQUEST_H class AbstractRequest { // public: virtual void setValue(int value) = 0; virtual void process() = 0; virtual void finish() = 0; }; #endif This class defines an API with three functions which a deriving class always has to implement, of which the process() and finish() functions are the most generic and likely to be used in any practical implementation. The setValue() function is specific to this demonstration implementation and would likely be adapted or extended to fit a real-life scenario. The advantage of using an abstract class as the basis for a request is that it allows the Dispatcher class to handle many different types of requests, as long as they all adhere to this same basic API. Using this abstract interface, we implement a basic Request class: #pragma once #ifndef REQUEST_H #define REQUEST_H #include "abstract_request.h" #include <string> using namespace std; typedef void (*logFunction)(string text); class Request : public AbstractRequest { int value; logFunction outFnc; public: void setValue(int value) { this->value = value; } void setOutput(logFunction fnc) { outFnc = fnc; } void process(); void finish(); }; #endif In its header file we first define the logging function pointer's format. After this we implement the request API, adding the setOutput() function to the base API, which accepts a function pointer for logging. Both setter functions merely assign the provided parameter to their respective private class members. Next, the class function implementations: #include "request.h" void Request::process() { outFnc("Starting processing request " + std::to_string(value) + "..."); // } void Request::finish() { outFnc("Finished request " + std::to_string(value)); }Both of these implementations are very basic, merely using the function pointer to output a string indicating the status of the worker thread. In a practical implementation, one would add the business logic to the process()function, with the finish() function containing any functionality to finish up a request, such as writing a map into a string. Worker class Next, the Worker class. This contains the logic which will be called by the dispatcher in order to process a request: #pragma once #ifndef WORKER_H #define WORKER_H #include "abstract_request.h" #include <condition_variable> #include <mutex> using namespace std; class Worker { condition_variable cv; mutex mtx; unique_lock<mutex> ulock; AbstractRequest* request; bool running; bool ready; public: Worker() { running = true; ready = false; ulock = unique_lock<mutex>(mtx); } void run(); void stop() { running = false; } void setRequest(AbstractRequest* request) { this->request = request; ready = true; } void getCondition(condition_variable* &cv); }; #endif Whereas the adding of a request to the dispatcher does not require any special logic, the Worker class does require the use of condition variables to synchronize itself with the dispatcher. For the C++11 threads API, this requires a condition variable, a mutex and a unique lock. The unique lock encapsulates the mutex and will ultimately be used with the condition variable as we will see in a moment. Beyond this we define methods to start and stop the worker, to set a new request for processing and to obtain access to its internal condition variable. Moving on, the rest of the implementation: #include "worker.h" #include "dispatcher.h" #include <chrono> using namespace std; void Worker::getCondition(condition_variable* &cv) { cv = &(this)->cv; } void Worker::run() { while (running) { if (ready) { ready = false; request->process(); request->finish(); } if (Dispatcher::addWorker(this)) { // Use the ready loop to deal with spurious wake-ups. while (!ready && running) { if (cv.wait_for(ulock, chrono::seconds(1)) == cv_status::timeout) { // We timed out, but we keep waiting unless // the worker is // stopped by the dispatcher. } } } } } Beyond the getter function for the condition variable, we define the run() function, which the dispatcher will run for each worker thread upon starting it. Its main loop merely checks that the stop() function hasn't been called yet, which would have set the running boolean value to false and ended the work thread. This is used by the dispatcher when shutting down, allowing it to terminate the worker threads. Since boolean values are generally atomic, setting and checking can be done simultaneously without risk or requiring a mutex. Moving on, the check of the ready variable is to ensure that a request is actually waiting when the thread is first run. On the first run of the worker thread, no request will be waiting and thus attempting to process one would result in a crash. Upon the dispatcher setting a new request, this boolean variable will be set to true. If a request is waiting, the ready variable will be set to false again, after which the request instance will have its process() and finish() functions called. This will run the business logic of the request on the worker thread's thread and finalize it. Finally, the worker thread adds itself to the dispatcher using its static addWorker() function. This function will return false if no new request was available, causing the worker thread to wait until a new request has become available. Otherwise the worker thread will continue with the processing of the new request that the dispatcher will have set on it. If asked to wait, we enter a new loop which will ensure that upon waking up from waiting for the condition variable to be signaled, we woke up because we got signaled by the dispatcher (ready variable set to true), and not because of a spurious wake-up. Last of all, we enter the actual wait() function of the condition variable, using the unique lock instance we created before, along with a timeout. If a timeout occurs, we can either terminate the thread, or keep waiting. Here we choose to do nothing and just re-enter the waiting loop. Dispatcher As the last item, we have the Dispatcher class itself: #pragma once #ifndef DISPATCHER_H #define DISPATCHER_H #include "abstract_request.h" #include "worker.h" #include <queue> #include <mutex> #include <thread> #include <vector> using namespace std; class Dispatcher { static queue<AbstractRequest*> requests; static queue<Worker*> workers; static mutex requestsMutex; static mutex workersMutex; static vector<Worker*> allWorkers; static vector<thread*> threads; public: static bool init(int workers); static bool stop(); static void addRequest(AbstractRequest* request); static bool addWorker(Worker* worker); }; #endif Most of this should look familiar by now. As one should have surmised by now, this is a fully static class. Moving on with its implementation: #include "dispatcher.h" #include <iostream> using namespace std; queue<AbstractRequest*> Dispatcher::requests; queue<Worker*> Dispatcher::workers; mutex Dispatcher::requestsMutex; mutex Dispatcher::workersMutex; vector<Worker*> Dispatcher::allWorkers; vector<thread*> Dispatcher::threads; bool Dispatcher::init(int workers) { hread* t = 0; Worker* w = 0; for (int i = 0; i < workers; ++i) { w = new Worker; allWorkers.push_back(w); = new thread(&Worker::run, w); hreads.push_back(t); } } After setting up the static class members, the init() function is defined. It starts the specified number of worker threads, keeping a reference to each worker and thread instance in their respective vector data structures. bool Dispatcher::stop() { for (int i = 0; i < allWorkers.size(); ++i) { allWorkers[i]->stop(); } cout <<"Stopped workers.n"; for (int j = 0; j < threads.size(); ++j) { hreads[j]->join(); cout <<"Joined threads.n"; } } In the stop() function each worker instance has its stop() function called. This will cause each worker thread to terminate, as we saw earlier in the Worker class description. Finally, we wait for each thread to join (that is, finish), prior to returning. void Dispatcher::addRequest(AbstractRequest* request) { workersMutex.lock(); if (!workers.empty()) { Worker* worker = workers.front(); worker->setRequest(request); condition_variable* cv; worker->getCondition(cv); cv->notify_one(); workers.pop(); workersMutex.unlock(); } else { workersMutex.unlock(); requestsMutex.lock(); requests.push(request); requestsMutex.unlock(); } } The addRequest() function is where things get interesting. In this one function a new request is added. What happens next to it depends on whether a worker thread is waiting for a new request or not. If no worker thread is waiting (worker queue is empty), the request is added to the request queue. The use of mutexes ensures that the access to these queues occurs safely, as the worker threads will simultaneously try to access both queues as well. An import gotcha to note here is the possibility of a deadlock. That is, a situation where two threads will hold the lock on a resource, with the other thread waiting for the first thread to release its lock before releasing its own. Every situation where more than one mutex is used in a single scope holds this potential. In this function the potential for deadlock lies in the releasing of the lock on the workers mutex and when the lock on the requests mutex is obtained. In the case that this function holds the workers mutex and tries to obtain the requests lock (when no worker thread is available), there is a chance that another thread holds the requests mutex (looking for new requests to handle), while simultaneously trying to obtain the workers mutex (finding no requests and adding itself to the workers queue). The solution hereby is simple: release a mutex before obtaining the next one. In the situation where one feels that more than one mutex lock has to be held it is paramount to examine and test one's code for potential deadlocks. In this particular situation the workers mutex lock is explicitly released when it is no longer needed, or before the requests mutex lock is obtained, preventing a deadlock. Another important aspect of this particular section of code is the way it signals a worker thread. As one can see in the first section of the if/else block, when the workers queue is not empty, a worker is fetched from the queue, has the request set on it and then has its condition variable referenced and signaled, or notified. Internally the condition variable uses the mutex we handed it before in the Worker class definition to guarantee only atomic access to it. When the notify_one() function (generally called signal() in other APIs) is called on the condition variable, it will notify the first thread in the queue of threads waiting for the condition variable to return and continue. In the Worker class'run() function we would be waiting for this notification event. Upon receiving it, the worker thread would continue and process the new request. The thread reference will then be removed from the queue until it adds itself again once it is done processing the request. bool Dispatcher::addWorker(Worker* worker) { bool wait = true; requestsMutex.lock(); if (!requests.empty()) { AbstractRequest* request = requests.front(); worker->setRequest(request); requests.pop(); wait = false; requestsMutex.unlock(); } else { requestsMutex.unlock(); workersMutex.lock(); workers.push(worker); workersMutex.unlock(); } return wait; } With this function a worker thread will add itself to the queue once it is done processing a request. It is similar to the earlier function in that the incoming worker is first actively matched with any request which may be waiting in the request queue. If none are available, the worker is added to the worker queue. Important to note here is that we return a boolean value which indicates whether the calling thread should wait for a new request, or whether it already has received a new request while trying to add itself to the queue. While this code is less complex than that of the previous function, it still holds the same potential deadlock issue due to the handling of two mutexes within the same scope. Here, too, we first release the mutex we hold before obtaining the next one. Makefile The Makefile for this dispatcher example is very basic again, gathering all C++ source files in the current folder and compiling them into a binary using g++: GCC := g++ OUTPUT := dispatcher_demo SOURCES := $(wildcard *.cpp) CCFLAGS := -std=c++11 -g3 all: $(OUTPUT) $(OUTPUT): $(GCC) -o $(OUTPUT) $(CCFLAGS) $(SOURCES) clean: rm $(OUTPUT) .PHONY: all Output After compiling the application, running it produces the following output for the fifty total requests: $ ./dispatcher_demo.exe Initialised. Starting processing request 1... Starting processing request 2... Finished request 1 Starting processing request 3... Finished request 3 Starting processing request 6... Finished request 6 Starting processing request 8... Finished request 8 Starting processing request 9... Finished request 9 Finished request 2 Starting processing request 11... Finished request 11 Starting processing request 12... Finished request 12 Starting processing request 13... Finished request 13 Starting processing request 14... Finished request 14 Starting processing request 7... Starting processing request 10... Starting processing request 15... Finished request 7 Finished request 15 Finished request 10 Starting processing request 16... Finished request 16 Starting processing request 17... Starting processing request 18... Starting processing request 0… At this point we we can already clearly see that even with each request taking almost no time to process, the requests are clearly being executed in parallel. The first request (request 0) only starts being processed after the 16th request, while the second request already finishes after the ninth request, long before this. The factors which determine which thread and thus which request is processed first depends on the OS scheduler and hardware-based scheduling. This clearly shows just how few assumptions one can be made about how a multithreaded application will be executed, even on a single platform. Starting processing request 5... Finished request 5 Starting processing request 20... Finished request 18 Finished request 20 Starting processing request 21... Starting processing request 4... Finished request 21 Finished request 4 Here the fourth and fifth requests also finish in a rather delayed fashion. Starting processing request 23... Starting processing request 24... Starting processing request 22... Finished request 24 Finished request 23 Finished request 22 Starting processing request 26... Starting processing request 25... Starting processing request 28... Finished request 26 Starting processing request 27... Finished request 28 Finished request 27 Starting processing request 29... Starting processing request 30... Finished request 30 Finished request 29 Finished request 17 Finished request 25 Starting processing request 19... Finished request 0 At this point the first request finally finishes. This may indicate that the initialization time for the first request will always delay it relative to the successive requests. Running the application multiple times can confirm this. It's important that if the order of processing is relevant, that this randomness does not negatively affect one's application. Starting processing request 33... Starting processing request 35... Finished request 33 Finished request 35 Starting processing request 37... Starting processing request 38... Finished request 37 Finished request 38 Starting processing request 39... Starting processing request 40... Starting processing request 36... Starting processing request 31... Finished request 40 Finished request 39 Starting processing request 32... Starting processing request 41... Finished request 32 Finished request 41 Starting processing request 42... Finished request 31 Starting processing request 44... Finished request 36 Finished request 42 Starting processing request 45... Finished request 44 Starting processing request 47... Starting processing request 48... Finished request 48 Starting processing request 43... Finished request 47 Finished request 43 Finished request 19 Starting processing request 34... Finished request 34 Starting processing request 46... Starting processing request 49... Finished request 46 Finished request 49 Finished request 45 Request 19 also became fairly delayed, showing once again just how unpredictable a multithreaded application can be. If we were processing a large data set in parallel here, with chunks of data in each request, we might have to pause at some points to account for these delays as otherwise our output cache might grow too large. As doing so would negatively affect an application's performance, one might have to look at low-level optimizations, as well as the scheduling of threads on specific processor cores in order to prevent this from happening. Stopped workers. Joined threads. Joined threads. Joined threads. Joined threads. Joined threads. Joined threads. Joined threads. Joined threads. Joined threads. Joined threads. Clean-up done. All ten worker threads which were launched in the beginning terminate here as we call the stop() function of the Dispatcher. Sharing data In this article's example we saw how to share information between threads in addition to the synchronizing of threads. This in the form of the requests we passed from the main thread into the dispatcher, from which each request gets passed on to a different thread. The essential idea behind the sharing of data between threads is that the data to be shared exists somewhere in a way which is accessible to two threads or more. After this we have to ensure that only one thread can modify the data, and that the data does not get modified while it's being read. Generally we would use mutexes or similar to ensure this. Using R/W-locks Readers-writer locks are a possible optimization here, because they allow multiple threads to read simultaneously from a single data source. If one has an application in which multiple worker threads read the same information repeatedly, it would be more efficient to use read-write locks than basic mutexes, because the attempts to read the data will not block the other threads. A read-write lock can thus be used as a more advanced version of a mutex, namely as one which adapts its behavior to the type of access. Internally it builds on mutexes (or semaphores) and condition variables. Using shared pointers First available via the Boost library and introduced natively with C++11, shared pointers are an abstraction of memory management using reference counting for heap-allocated instances. They are partially thread-safe, in that creating multiple shared pointer instances can be created, but the referenced object itself is not thread-safe. Depending on the application this may suffice, however. To make them properly thread-safe one can use atomics. Summary In this article we looked at how to pass data between threads in a safe manner as part of a fairly complex scheduler implementation. We also looked at the resulting asynchronous processing of said scheduler and considered some potential alternatives and optimizations for passing data between threads. At this point one should be able to safely pass data between threads, as well as synchronize the access to other shared resources. In the next article we will be looking at the native C++ threading and primitives API.  Resources for Article: Further resources on this subject: Multithreading with Qt [article] Understanding the Dependencies of a C++ Application [article] Boost.Asio C++ Network Programming [article]

(For more resources related to this topic, see here.) PWM allows us to drive a certain pin to on and off at desired frequency and duty cycle. This way we can pulse our LEDs much faster than our eyes can react, and while we only see a dimmer or a brighter LED, if one would look at it with a high speed camera, one would see that the LED still only turns on or off. Our eyes just perceive this differently. There are three basic terms you need to understand about PWM: Frequency: This defines how many full on/off "cycles" are generated in a second Period: This defines how long one complete pulse cycle takes Duty cycle: This specifies the time the signal is high during one full period Think about the following figure: In the preceding figure, we have PWM generating first a 50 percent duty cycle and then dropping to 25 percent. The effective time the LED spends as on and off can be controlled with very high precision, and this way we can achieve smooth brightness fluctuations. Let's try doing just that. First, we will design a schematic with two LEDs that are connected to two different PWMs. Nothing fancy here either really, we have a current limiting resistor after the LEDs and that's it. This time we will be using input from both headers, as PWM1 and PWM2 are located on P9 and P8, respectively. Our third and last program in this article will be called racing_PWMs.py. As usual, we need to include the Adafruit library here as well, this time the PWM part of it: import Adafruit_BBIO.PWM as PWM When you initialize a PWM, you can initialize it with two to four parameters listed as follows: Channel (header pin) Duty (as in percent, 0-100) Frequency (periods in a second, default is 2000) And polarity (0 or 1. With this you can invert the duty cycle, default is 0) So, we will initialize both our channels: PWM.start("P9_14", 50, 100) #50% duty and 100hz cycle PWM.start("P8_13", 50, 100) At this point, the LEDs will light up, and now we can start changing our duty cycle in a loop to adjust the average voltage. Full listing of racing_PWMs.py is as follows: #!/usr/bin/python import time import Adafruit_BBIO.PWM as PWM sleep_time=0.005 #The lower the value the faster the activity PWM.start("P9_14", 50, 100) #50% duty and 100hz cycle PWM.start("P8_13", 50, 100) while True: for i in range(100,1, -1): PWM.set_duty_cycle("P9_14", i) # Dimming PWM.set_duty_cycle("P8_13", abs(i-100)+1) # Getting brighter time.sleep(sleep_time) for i in range(1, 100): PWM.set_duty_cycle("P9_14", i) PWM.set_duty_cycle("P8_13", abs(i-100)+1 ) time.sleep(sleep_time) When you run the program, you should see both LEDs racing (blinking) in opposite phases. As you see, the Adafruit BBIO library is extremely useful and easy to use. And so far we have only used two functionalities it provides. Actually, the library also supports easy access to SPI and I2C communication and Analog to Digital Converter(ADC) as well. Summary In this article we went through foundations of input and output on a very basic level. We talked about the general purpose I/O pins, and how they can be used to control external components, such as LEDs or buttons. As you must have gathered already, with proper application of voltage and care, we can operate many of the basic electronic components. You should now feel comfortable with the basic building blocks that can help you build some external inputs and outputs to your Beagle programs. Resources for Article: Further resources on this subject: Home Security by BeagleBone [Article] Introducing BeagleBoard [Article] Viewing on Mobile Devices [Article]

Yesterday, Microsoft announced the release of TypeScript 3.7 with new tooling features, optional chaining, nullish coalescing, assertion functions, and much more. This release also includes breaking features; a few changes in the DOM where the types in lib.dom.d.ts have been updated; the typeArguments property has been removed from the TypeReference interface. Also, TypeScript 3.7 emits get/set accessors in .d.ts files which can cause breaking changes for consumers on older versions of TypeScript like 3.5 and prior. TypeScript 3.6 users will not be impacted as the version was future-proofed for this feature. Let us have a look at other new features in TypeScript 3.7. What’s new in TypeScript 3.7? Optional Chaining TypeScript 3.7 implements Optional Chaining, one of the most highly-demanded ECMAScript features that was filed 5 years ago. Optional chaining lets one write code that can immediately stop running some expressions if it is run into a null or undefined. The star of the show in optional chaining is the new ?. operator for optional property accesses. Optional chaining also includes two other operations; optional element access, which acts similarly to optional property accesses, but allows us to access non-identifier properties (e.g. arbitrary strings, numbers, and symbols). The second one is an optional call, which allows to conditionally call expressions if they’re not null or undefined. Assertion Functions Assertion functions are a specific set of functions that throw an error if something unexpected happens. Assertions in JavaScript are often used to guard against improper types being passed in. Unfortunately in TypeScript, these checks could never be properly encoded. For loosely-typed code, this meant TypeScript was checking less, and for slightly conservative code it often forced users to use type assertions. Another alternative was to rewrite the code such that the language could analyze it. However, this was not convenient. To solve this, TypeScript 3.7 introduces a new concept called “assertion signatures” which models these assertion functions. The first type of assertion signature ensures that whatever condition is being checked must be true for the remainder of the containing scope. The other type of assertion signature doesn’t check for a condition but instead tells TypeScript that a specific variable or property has a different type. Build-Free Editing with Project References In TypeScript 3.7, when opening a project with dependencies, TypeScript will automatically use the source .ts/.tsx files instead. This means projects using project references will now see an improved editing experience where semantic operations are up-to-date. Website and Playground Updates TypeScript playground now includes awesome new features like quick fixes to fix errors, dark/high-contrast mode, and automatic type acquisition so you can import other packages. Each feature here is explained through interactive code snippets under the “what’s new” menu. Many users and developers are excited to try out TypeScript 3.7. https://twitter.com/kmsaldana1/status/1191768934648729600 https://twitter.com/mgechev/status/1191769805952438272 To know more about other new features in TypeScript 3.7, read the official release notes. Announcing Feathers 4, a framework for real-time apps and REST APIs with JavaScript or TypeScript Microsoft introduces Static TypeScript, as an alternative to embedded interpreters, for programming MCU-based devices TypeScript 3.6 releases with stricter generators, new functions in TypeScript playground, better Unicode support for identifiers, and more

Microservices have been growing in popularity since the past few years, 2014 to be precise. Honestly speaking they weren’t that popular until around 2016 - take a look at the steep rise in the curve. The outbreak has happened over the past few years and there are quite a few factors contributing to their growth, like the cloud, distributed architectures, etc. Source: Google Trends Microservices allow for a clearer and refined architecture, with services built to work in isolation, without affecting the resilience and robustness of the application in any way. But does that mean that the Monolith is dead and only Microservices reign? Let’s find out, shall we? Those of you who participated in this year’s survey, I thank you for taking the time out to share such valuable information. For those of you who don’t know what the survey is all about, it a thing that we do every year, where thousands of developers, architects, managers, admins, share their insights with us, and we share our findings with the community. This year’s survey was as informative as the last, if not more! We had developers tell us so much about what they’re doing, where they see technology heading and what tools and techniques they use to stay relevant at what they do. So we took the opportunity and asked our respondents a question about the topic under discussion. Source: WWE.com Revelations If I asked a developer in 2018, what they thought would be the response, they’d instantly say that a majority would be for microservices. Source: Packtpub Skill Up Survey 2018 If you were the one who guessed the answer was going to be Yes, give yourself a firm pat on the back! It’s great to see that 1,603 people are throwing their hands up in the air and building microservices. On the other hand, it’s possible that it’s purely their manager’s decision (See how this forms a barrier to achieving business goals). Anyway, I was particularly concerned about the remaining 314 people who said ‘No’ (those who skipped answering, now is your chance to say something in the comments section below!). Why no Microservices? I thought I’d analyse the possibilities as to why one wouldn’t want to use the microservices pattern in their application architecture. It’s not like developers are migrating from monoliths to microservices, just because everyone else is doing it. Like any other architectural decision, there are several factors that need to be taken into consideration before making the switch. So here’s what I thought were some reasons why developers are sticking to monoliths. #1 One troll vs many elves: Complex times Well imagine you could be attacked by one troll or a hundred house elves. Which situation would you choose to be in if neither isn’t an option? I don’t know about you, but I’d choose the troll any day! Keeping the troll’s size aside, I’d be better off knowing I had one large enemy in front of me, rather than being surrounded by a hundred miniature ones. The same goes for microservices. More services means more complexity, more issues that could crop up. For developers, more services means that they would need to run or connect to all of them on their machine. Although there are tools that help solve this problem, you have to admit that it’s a task to run all services together as a whole application. On the other hand, Ops professionals are tasked to monitor and keep all these services up and running. #2 We lack the expertise Let alone having Developer Rockstars or Admin Ninjas (Oops, I shouldn’t be using those words now, find out why), if your organisation lacks experienced professionals, you’ve got a serious problem. What if there’s an organisation that has been having issues developing/managing a monolith itself. There’s no guarantee that they will be able to manage a microservices based application more effectively. It’s a matter of the organisation having enough hands on skills needed to perform these tasks. These skills are tough to acquire and it’s not simple for organisations to find the right talent. #3 Tower of Babel: Communication gaps In a monolith, communication happens within the application itself and the network channels exist internally. However, this isn’t the case for a microservices architecture as inter-service communication is necessary to keep everything running in tandem. This results in the generation of multiple points of failure, complicating things. To minimise failure, each service has a certain number of retries when trying to establish communication with another. When scaled up, these retries add a load on the database, what with communication formats having to follow strict rules to avoid complexity back again. It’s a vicious circle! #4 Rebuilding a monolith When you build an application based on the microservices architecture, you may benefit a great deal from robustness and reliability. However, microservices together form a large, complicated system, which can be managed by orchestration platforms like Kubernetes. Although, if individual teams are managing clusters of these services, it’s quite likely that orchestration, deployment and management of such a system will be a pain. #5 Burning in dependency hell Microservices are notorious for inviting developers to build services in various languages and then to glue them together. While this is an advantage to a certain extent, it complicates dependency management in the entire application. Moreover, dependencies get even more complicated when versions of tools don’t receive instantaneous support as they are updated. You and your team can go crazy keeping track of versions and dependencies that need to be managed to maintain smooth functioning of your application. So while the microservice architecture is hot, it is not always the best option and teams can actually end up making things worse if they choose to make the change unprepared. Yes, the cloud does benefit much more when applications are deployed as services, rather than as a monolith, but the renowned/infamous “lift and shift” method still exists and works when needed. Ultimately, if you think past the hype, the monolith is not really dead yet and is in fact still being deployed and run in several organisations. Finally, I want to stress that it’s critical that developers and architects take a well informed decision, keeping in mind all the above factors, before they choose an architecture. Like they say, “With great power comes great responsibility”, that’s exactly what great architecture is all about, rather than just jumping on the bandwagon. Building Scalable Microservices Why microservices and DevOps are a match made in heaven What is a multi layered software architecture?

You might think that Rust is only meant to be used for complex system development, or that it should be used where security is the number one concern. Thinking of using it for web development might sound to you like huge overkill. We already have proven web-oriented languages that have worked until now, such as PHP or JavaScript, right? This is far from true. Many projects use the web as their platform and for them, it's sometimes more important to be able to receive a lot of traffic without investing in expensive servers rather than using legacy technologies, especially in new products. This is where Rust comes in handy. Thanks to its speed and some really well thought out web-oriented frameworks, Rust performs even better than the legacy web programming languages. In this tutorial, we'll see how Rust can be used for Web Development. This article is an extract from Rust High Performance, authored by Iban Eguia Moraza. Rust is even trying to replace some of the JavaScript on the client side of applications, since Rust can compile to WebAssembly, making it extremely powerful for heavy client-side web workloads. Creating extremely efficient web templates We have seen that Rust is a really efficient language and metaprogramming allows for the creation of even more efficient code. Rust has great templating language support, such as Handlebars and Tera. Rust's Handlebars implementation is much faster than the JavaScript implementation, while Tera is a template engine created for Rust based on Jinja2. In both cases, you define a template file and then you use Rust to parse it. Even though this will be reasonable for most web development, in some cases, it might be slower than pure Rust alternatives. This is where the Maud crate comes in. We will see how it works and how it achieves orders of magnitude faster performance than its counterparts. To use Maud, you will need nightly Rust, since it uses procedural macros. As we saw in previous chapters, if you are using rustup you can simply run rustup override set nightly. Then, you will need to add Maud to your Cargo.toml file in the [dependencies] section: [dependencies] maud = "0.17.2 Maud brings an html!{} procedural macro that enables you to write HTML in Rust. You will, therefore, need to import the necessary crate and macro in your main.rs or lib.rs file, as you will see in the following code. Remember to also add the procedural macro feature at the beginning of the crate: #![feature(proc_macro)] extern crate maud; use maud::html; You will now be able to use the html!{} macro in your main() function. This macro will return a Markup object, which you can then convert to a String or return to Rocket or Iron for your website implementation (you will need to use the relevant Maud features in that case). Let's see what a short template implementation looks like: fn main() { use maud::PreEscaped; let user_name = "FooBar"; let markup = html! { (PreEscaped("<!DOCTYPE html>")) html { head { title { "Test website" } meta charset="UTF-8"; } body { header { nav { ul { li { "Home" } li { "Contact Us" } } } } main { h1 { "Welcome to our test template!" } p { "Hello, " (user_name) "!" } } footer { p { "Copyright © 2017 - someone" } } } } }; println!("{}", markup.into_string()); } It seems like a complex template, but it contains just the basic information a new website should have. We first add a doctype, making sure it will not escape the content (that is what the PreEscaped is for) and then we start the HTML document with two parts: the head and the body. In the head, we add the required title and the charset meta element to tell the browser that we will be using UTF-8. Then, the body contains the three usual sections, even though this can, of course, be modified. One header, one main section, and one footer. I added some example information in each of the sections and showed you how to add a dynamic variable in the main section inside a paragraph. The interesting syntax here is that you can create elements with attributes, such as the meta element, even without content, by finishing it early with a semicolon. You can use any HTML tag and add variables. The generated code will be escaped, except if you ask for non-escaped data, and it will be minified so that it occupies the least space when being transmitted. Inside the parentheses, you can call any function or variable that returns a type that implements the Display trait and you can even add any Rust code if you add braces around it, with the last statement returning a Display element. This works on attributes too. This gets processed at compile time, so that at runtime it will only need to perform the minimum possible amount of work, making it extremely efficient. And not only that; the template will be typesafe thanks to Rust's compile-time guarantees, so you won't forget to close a tag or an attribute. There is a complete guide to the templating engine that can be found at https://maud.lambda.xyz/. Connecting with a database If we want to use SQL/relational databases in Rust, there is no other crate to think about than Diesel. If you need access to NoSQL databases such as Redis or MongoDB, you will also find proper crates, but since the most used databases are relational databases, we will check Diesel here. Diesel makes working with MySQL/MariaDB, PostgreSQL, and SQLite very easy by providing a great ORM and typesafe query builder. It prevents all potential SQL injections at compile time, but is still extremely fast. In fact, it's usually faster than using prepared statements, due to the way it manages connections to databases. Without entering into technical details, we will check how this stable framework works. The development of Diesel has been impressive and it's already working in stable Rust. It even has a stable 1.x version, so let's check how we can map a simple table. Diesel comes with a command-line interface program, which makes it much easier to use. To install it, run cargo install diesel_cli. Note that, by default, this will try to install it for PostgreSQL, MariaDB/MySQL, and SQLite. For this short tutorial, you need to have SQLite 3 development files installed, but if you want to avoid installing all MariaDB/MySQL or PostgreSQL files, you should run the following command: cargo install --no-default-features --features sqlite diesel_cli Then, since we will be using SQLite for our short test, add a file named .env to the current directory, with the following content: DATABASE_URL=test.sqlite We can now run diesel setup and diesel migration generate initial_schema. This will create the test.sqlite SQLite database and a migrations folder, with the first empty initial schema migration. Let's add this to the initial schema up.sql file: CREATE TABLE 'users' ( 'username' TEXT NOT NULL PRIMARY KEY, 'password' TEXT NOT NULL, 'email' TEXT UNIQUE ); In its counterpart down.sql file, we will need to drop the created table: DROP TABLE `users`; Then, we can execute diesel migration run and check that everything went smoothly. We can execute diesel migration redo to check that the rollback and recreation worked properly. We can now start using the ORM. We will need to add diesel, diesel_infer_schema, and dotenv to our Cargo.toml. The dotenv crate will read the .env file to generate the environment variables. If you want to avoid using all the MariaDB/MySQL or PostgreSQL features, you will need to configure diesel for it: [dependencies] dotenv = "0.10.1" [dependencies.diesel] version = "1.1.1" default-features = false features = ["sqlite"] [dependencies.diesel_infer_schema] version = "1.1.0" default-features = false features = ["sqlite"] Let's now create a structure that we will be able to use to retrieve data from the database. We will also need some boilerplate code to make everything work: #[macro_use] extern crate diesel; #[macro_use] extern crate diesel_infer_schema; extern crate dotenv; use diesel::prelude::*; use diesel::sqlite::SqliteConnection; use dotenv::dotenv; use std::env; #[derive(Debug, Queryable)] struct User { username: String, password: String, email: Option<String>, } fn establish_connection() -> SqliteConnection { dotenv().ok(); let database_url = env::var("DATABASE_URL") .expect("DATABASE_URL must be set"); SqliteConnection::establish(&database_url) .expect(&format!("error connecting to {}", database_url)) } mod schema { infer_schema!("dotenv:DATABASE_URL"); } Here, the establish_connection() function will call dotenv() so that the variables in the .env file get to the environment, and then it uses that DATABASE_URL variable to establish the connection with the SQLite database and returns the handle. The schema module will contain the schema of the database. The infer_schema!() macro will get the DATABASE_URL variable and connect to the database at compile time to generate the schema. Make sure you run all the migrations before compiling. We can now develop a small main() function with the basics to list all of the users from the database: fn main() { use schema::users::dsl::*; let connection = establish_connection(); let all_users = users .load::<User>(&connection) .expect("error loading users"); println!("{:?}", all_users); } This will just load all of the users from the database into a list. Notice the use statement at the beginning of the function. This retrieves the required information from the schema for the users table so that we can then call users.load(). As you can see in the guides at diesel.rs, you can also generate Insertable objects, which might not have some of the fields with default values, and you can perform complex queries by filtering the results in the same way you would write a SELECT statement. Creating a complete web server There are multiple web frameworks for Rust. Some of them work in stable Rust, such as Iron and Nickel Frameworks, and some don't, such as Rocket. We will talk about the latter since, even if it forces you to use the latest nightly branch, it's so much more powerful than the rest that it really makes no sense to use any of the others if you have the option to use Rust nightly. Using Diesel with Rocket, apart from the funny wordplay joke, works seamlessly. You will probably be using the two of them together, but in this section, we will learn how to create a small Rocket server without any further complexity. There are some boilerplate code implementations that add a database, cache, OAuth, templating, response compression, JavaScript minification, and SASS minification to the website, such as my Rust web template in GitHub if you need to start developing a real-life Rust web application. Rocket trades that nightly instability, which will break your code more often than not, for simplicity and performance. Developing a Rocket application is really easy and the performance of the results is astonishing. It's even faster than using some other, seemingly simpler frameworks, and of course, it's much faster than most of the frameworks in other languages. So, how does it feel to develop a Rocket application? We start by adding the latest rocket and rocket_codegen crates to our Cargo.toml file and adding a nightly override to our current directory by running rustup override set nightly. The rocket crate contains all the code to run the server, while the rocket_codegen crate is actually a compiler plugin that modifies the language to adapt it for web development. We can now write the default Hello, world! Rocket example: #![feature(plugin)] #![plugin(rocket_codegen)] extern crate rocket; #[get("/")] fn index() -> &'static str { "Hello, world!" } fn main() { rocket::ignite().mount("/", routes![index]).launch(); } In this example, we can see how we ask Rust to let us use plugins to then import the rocket_codegen plugin. This will enable us to use attributes such as #[get] or #[post] with request information that will generate boilerplate code when compiled, leaving our code fairly simple for our development. Also, note that this code has been checked with Rocket 0.3 and it might fail in a future version, since the library is not stable yet. In this case, you can see that the index() function will respond to any GET request with a base URL. This can be modified to accept only certain URLs or to get the path of something from the URL. You can also have overlapping routes with different priorities so that if one is not taken for a request guard, the next will be tried. And, talking about request guards, you can create objects that can be generated when processing a request that will only let the request process a given function if they are properly built. This means that you can, for example, create a User object that will get generated by checking the cookies in the request and comparing them in a Redis database, only allowing the execution of the function for logged-in users. This easily prevents many logic flaws. The main() function ignites the Rocket and mounts the index route at /. This means that you can have multiple routes with the same path mounted at different route paths and they do not need to know about the whole path in the URL. In the end, it will launch the Rocket server and if you run it with cargo run, it will show the following: If you go to the URL, you will see the Hello, World! message. Rocket is highly configurable. It has a rocket_contrib crate which offers templates and further features, and you can create responders to add GZip compression to responses. You can also create your own error responders when an error occurs. You can also configure the behavior of Rocket by using the Rocket.toml file and environment variables. As you can see in this last output, it is running in development mode, which adds some debugging information. You can configure different behaviors for staging and production modes and make them perform faster. Also, make sure that you compile the code in --release mode in production. If you want to develop a web application in Rocket, make sure you check https://rocket.rs/ for further information. Future releases also look promising. Rocket will implement native CSRF and XSS prevention, which, in theory, should prevent all XSS and CSRF attacks at compile time. It will also make further customizations to the engine possible. If you found this article useful and would like to learn more such tips, head over to pick up the book, Rust High Performance, authored by Iban Eguia Moraza. Mozilla is building a bridge between Rust and JavaScript Perform Advanced Programming with Rust Say hello to Sequoia: a new Rust based OpenPGP library to secure your apps

Dive deeper into the world of AI innovation and stay ahead of the AI curve! Subscribe to our AI_Distilled newsletter for the latest insights and books. Don't miss out – sign up today!Introduction As datasets continue growing rapidly in size and complexity, exploring, preparing, and documenting data is taking data scientists more and more time. While coding is crucial for actually implementing analyses, there's an opportunity to make the conceptual work more efficient through a simpler way of interaction.ChatGPT - an AI tool that truly understands what you're saying and can have natural back-and-forth conversations. By casually describing what you need to do, its broad knowledge lets it generate sample results, code snippets, and documentation without you writing a single line.In this fast-paced world where time is precious, ChatGPT seems like a helpful extra pair of hands. You can bounce ideas off it 24/7 to test hypotheses and get artifacts to aid your own work.Now, it definitely won't take over your job. However, facilitating exploratory talks and quick prototyping through plain speech, it opens up new approaches that maximize our growing computing power.In this post, I'll demonstrate how ChatGPT streamlines common analyst tasks through example conversations. While coding is still king, it serves as a supplementary brain to speed up the often overlooked definition stage - arguably the most important part. Ultimately, combining human intuition with AI assistance will vastly improve our ability to extract value from huge datasets at lightning speed.Let's see how it can supplement analysis in practice.You can access the entire code here: https://github.com/rohanchikorde/chatgpt-for-data-analysis.gitCreating a Sample DataFrame using ChatGPTHere is a sample prompt to have ChatGPT generate a Pandas DataFrame containing fake customer data:Prompt:Hey, can you help me create a sample DataFrame containing customer data that I can use to demonstrate exploring and analyzing a dataset through conversations? I need it to contain around 10 rows with the following columns: ·        'CustomerID' (integer index) ·        'Name' (string of first and last name) ·        'Age' (integer) ·        'State' (string of two-letter state abbreviation) ·        'Annual Income' (integer dollar amount) You can generate random values for the names, ages between 20-60, state abbreviations from the 50 US states, and incomes between $30,000-$150,000. It would be great if you could print out the code to create a Pandas DataFrame with this sample data so I have something concrete to reference in example conversations. Having sample data upfront will make it easier to walk through tasks like summarizing, grouping, plotting and more.Viola! Here, is the generated Python code by ChatGPT:import pandas as pd import numpy as np # Define the number of rows num_rows = 10 # Set random seed for reproducibility np.random.seed(42) # Define the state abbreviations for the 50 US states state_abbreviations = ['AL', 'AK', 'AZ', 'AR', 'CA', 'CO', 'CT', 'DE', 'FL', 'GA',                       'HI', 'ID', 'IL', 'IN', 'IA', 'KS', 'KY', 'LA', 'ME', 'MD',                       'MA', 'MI', 'MN', 'MS', 'MO', 'MT', 'NE', 'NV', 'NH', 'NJ',                       'NM', 'NY', 'NC', 'ND', 'OH', 'OK', 'OR', 'PA', 'RI', 'SC',                       'SD', 'TN', 'TX', 'UT', 'VT', 'VA', 'WA', 'WV', 'WI', 'WY'] # Generate random sample data customer_data = {    'CustomerID': range(1, num_rows + 1),    'Name': [' '.join(np.random.choice(['John', 'Jane', 'David', 'Sarah', 'Michael', 'Emily'], 1)) +             ' ' +             np.random.choice(['Smith', 'Johnson', 'Williams', 'Jones', 'Brown', 'Davis'], 1)[0]            for _ in range(num_rows)],    'Age': np.random.randint(20, 61, num_rows),    'State': np.random.choice(state_abbreviations, num_rows),    'Annual Income': np.random.randint(30000, 150001, num_rows) } # Create the DataFrame df = pd.DataFrame(customer_data) # Print the DataFrame print(df) After running the above code:Descriptive StatisticsDescriptive statistics are a fundamental aspect of data analysis that provides a summary of the main characteristics of a dataset. They help us understand the distribution, central tendency, and variability of the data. Let's explore some common descriptive statistics and how they can be calculated and interpreted:Measures of Central Tendency:Mean: It represents the average value of a dataset and is computed by summing all the values and dividing by the number of observations.Median: It corresponds to the middle value of a dataset when it is sorted in ascending or descending order. It is less affected by extreme values compared to the mean.Mode: It is the most frequently occurring value in a dataset. Python Code by ChatGPT:import pandas as pd # Calculate the mean age_mean = df['Age'].mean() income_mean = df['Annual Income'].mean() # Calculate the median age_median = df['Age'].median() income_median = df['Annual Income'].median() # Calculate the mode age_mode = df['Age'].mode().values income_mode = df['Annual Income'].mode().values # Print the results print("Age Mean:", age_mean) print("Age Median:", age_median) print("Age Mode:", age_mode) print("Income Mean:", income_mean) print("Income Median:", income_median) print("Income Mode:", income_mode)Output in Python environment:Measures of Dispersion/VariabilityRange: It is the difference between the maximum and minimum values in a dataset, providing an idea of the spread of the data.Variance: It quantifies the average squared deviation of each data point from the mean. A higher variance indicates greater dispersion.Standard Deviation: It is the square root of the variance and provides a measure of the average distance between each data point and the mean. Python code generated by ChatGPT: import pandas as pd # Calculate the range age_range = df['Age'].max() - df['Age'].min() income_range = df['Annual Income'].max() - df['Annual Income'].min() # Calculate the variance age_variance = df['Age'].var() income_variance = df['Annual Income'].var() # Calculate the standard deviation age_std_dev = df['Age'].std() income_std_dev = df['Annual Income'].std() # Print the results print("Age Range:", age_range) print("Age Variance:", age_variance) print("Age Standard Deviation:", age_std_dev) print("Income Range:", income_range) print("Income Variance:", income_variance) print("Income Standard Deviation:", income_std_dev) Output in Python environment:PercentilesPercentiles divide a dataset into hundredths, allowing us to understand how values are distributed. The median corresponds to the 50th percentile.Quartiles divide the dataset into quarters, with the first quartile (Q1) representing the 25th percentile and the third quartile (Q3) representing the 75th percentile.Python code generated by ChatGPT:import pandas as pd # Calculate the percentiles age_percentiles = df['Age'].quantile([0.25, 0.5, 0.75]) income_percentiles = df['Annual Income'].quantile([0.25, 0.5, 0.75]) # Extract the quartiles age_q1, age_median, age_q3 = age_percentiles income_q1, income_median, income_q3 = income_percentiles # Print the results print("Age Percentiles:") print("Q1 (25th percentile):", age_q1) print("Median (50th percentile):", age_median) print("Q3 (75th percentile):", age_q3) print("\nIncome Percentiles:") print("Q1 (25th percentile):", income_q1) print("Median (50th percentile):", income_median) print("Q3 (75th percentile):", income_q3)Output in Python environment:Skewness and KurtosisSkewness measures the asymmetry of a distribution. A positive skew indicates a longer tail on the right, while a negative skew indicates a longer tail on the left.Kurtosis measures the heaviness of the tails of a distribution. High kurtosis implies more extreme values, while low kurtosis indicates a flatter distribution. Python Code generated by ChatGPT:import pandas as pd # Calculate the skewness age_skewness = df['Age'].skew() income_skewness = df['Annual Income'].skew() # Calculate the kurtosis age_kurtosis = df['Age'].kurtosis() income_kurtosis = df['Annual Income'].kurtosis() # Print the results print("Age Skewness:", age_skewness) print("Income Skewness:", income_skewness) print("\nAge Kurtosis:", age_kurtosis) print("Income Kurtosis:", income_kurtosis) Output in Python jupyter notebook:Grouping and AggregationGrouping and aggregation in Python are powerful techniques for analyzing data by grouping it based on specific criteria and calculating summary statistics or performing aggregate functions on each group. Here's the Python code to group the data by state and find the average age and income for each state:import pandas as pd # Group the data by State and calculate the average age and income grouped_data = df.groupby('State').agg({'Age': 'mean', 'Annual Income': 'mean'}) # Print the grouped data print(grouped_data) Output in Python jupyter notebook:In this code, ChatGPT uses the groupby function from the Pandas library to group the data in the DataFrame df by the 'State' column. It then uses the agg function to specify the aggregation functions we want to apply to each group. In this case, it calculates the mean of the 'Age' and 'Annual Income' columns for each state.The output of this code will be a new DataFrame containing the grouped data with the average age and income for each state. The DataFrame will have the 'State' column as the index and two additional columns: 'Age' and 'Annual Income', representing the average values for each state.Data VisualizationHistogram of AgeThe histogram provides a visual representation of the distribution of ages in the dataset. The x-axis represents the age values, and the y-axis represents the frequency or count of individuals falling into each age bin. The shape of the histogram can provide insights into the data's central tendency, variability, and any skewness in the distribution.Scatter Plot: Age vs. Annual IncomeThe scatter plot visualizes the relationship between age and annual income for each data point. Each point on the plot represents an individual's age and their corresponding annual income. By plotting the data points, we can observe patterns, clusters, or trends in the relationship between these two variables. The scatter plot helps identify any potential correlation or lack thereof between age and income.Python Code for histogram and scatterplot generated by ChatGPT:import matplotlib.pyplot as plt # Plot a histogram of the Age variable plt.hist(df['Age']) plt.xlabel('Age') plt.ylabel('Frequency') plt.title('Histogram of Age') plt.show() # Plot a scatter plot between Age and Income plt.scatter(df['Age'], df['Annual Income']) plt.xlabel('Age') plt.ylabel('Annual Income') plt.title('Scatter Plot: Age vs. Annual Income') plt.show() Output in Python jupyter notebook In this code, ChatGPT uses the hist function from the matplotlib library to plot a histogram of the 'Age' variable. The histogram visualizes the distribution of ages in the dataset. It set the x-axis label to 'Age', the y-axis label to 'Frequency' (indicating the count of individuals in each age group), and give the plot a title which is super cool.Next, it used the scatter function to create a scatter plot between 'Age' and 'Annual Income'. The scatter plot shows the relationship between age and annual income for each data point. It sets the x-axis label to 'Age', the y-axis label to 'Annual Income', and gives the plot a title.ConclusionIn this blog, we explored a couple of examples showing how ChatGPT can streamline various aspects of data analysis through natural conversation. By simply describing our needs, it was able to generate sample Python code for us without writing a single line of code. While the results require human review, ChatGPT handles much of the prototyping work rapidly.For data scientists who understand programming but want to focus more on problem definition, ChatGPT serves as a helpful digital assistant to offload some of the repetitive technical work. It also opens up analysis to those without coding skills by abstracting the process into simple question-and-response dialogue. While ChatGPT does not replace human expertise, it makes the analysis process more approachable and efficient overall.Going forward, as chatbots advance in capabilities, we may see them automating ever more complex portions of the data science lifecycle through natural language. But for now, even with its limitations, ChatGPT has proven quite useful as a dialogue-driven aid for getting initial insights, especially when time is of the essence. I hope this post demonstrates how accessible and powerful conversational data science can be.Author BioRohan Chikorde is an accomplished AI Architect professional with a post-graduate in Machine Learning and Artificial Intelligence. With almost a decade of experience, he has successfully developed deep learning and machine learning models for various business applications. Rohan's expertise spans multiple domains, and he excels in programming languages such as R and Python, as well as analytics techniques like regression analysis and data mining. In addition to his technical prowess, he is an effective communicator, mentor, and team leader. Rohan's passion lies in machine learning, deep learning, and computer vision.

In this article by Alex Libby, author of Learning Less.js, we'll see how Less can make creating media queries a cinch; we will cover the following topics: How media queries work What's wrong with CSS? Creating a simple example (For more resources related to this topic, see here.) Introducing media queries If you've ever spent time creating content for sites, particularly for display on a mobile platform, then you might have come across media queries. For those of you who are new to the concept, media queries are a means of tailoring the content that is displayed on screen when the viewport is resized to a smaller size. Historically, websites were always built at a static size—with more and more people viewing content on smartphones and tablets, this means viewing them became harder, as scrolling around a page can be a tiresome process! Thankfully, this became less of an issue with the advent of media queries—they help us with what should or should not be displayed when viewing content on a particular device. Almost all modern browsers offer native support for media queries—the only exception being IE Version 8 or below, where it is not supported natively: Media queries always begin with @media and consist of two parts: The first part, only screen, determines the media type where a rule should apply—in this case, it will only show the rule if we're viewing content on screen; content viewed when printed can easily be different. The second part, or media feature, (min-width: 530px) and (max-width: 949px), means the rule will only apply between a screen size set at a minimum of 530px and a maximum of 949px. This will rule out any smartphones and will apply to larger tablets, laptops, or PCs. There are literally dozens of combinations of media queries to suit a variety of needs—for some good examples, visit http://cssmediaqueries.com/overview.html, where you can see an extensive list, along with an indication whether it is supported in the browser you normally use. Media queries are perfect to dynamically adjust your site to work in multiple browsers—indeed, they are an essential part of a responsive web design. While browsers support media queries, there are some limitations we need to consider; let's take a look at these now. The limitations of CSS If we spend any time working with media queries, there are some limitations we need to consider; these apply equally if we were writing using Less or plain CSS: Not every browser supports media features uniformly; to see the differences, visit http://cssmediaqueries.com/overview.html using different browsers. Current thinking is that a range of breakpoints has to be provided; this can result in a lot of duplication and a constant battle to keep up with numerous different screen sizes! The @media keyword is not supported in IE8 or below; you will need to use JavaScript or jQuery to achieve the same result, or a library such as Modernizr to provide a graceful fallback option. Writing media queries will tie your design to a specific display size; this increases the risk of duplication as you might want the same element to appear in multiple breakpoints, but have to write individual rules to cover each breakpoint. Breakpoints are points where your design will break if it is resized larger or smaller than a particular set of given dimensions. The traditional thinking is that we have to provide different style rules for different breakpoints within our style sheets. While this is valid, ironically it is something we should not follow! The reason for this is the potential proliferation of breakpoint rules that you might need to add, just to manage a site. With care and planning and a design-based breakpoints mindset, we can often get away with a fewer number of rules. There is only one breakpoint given, but it works in a range of sizes without the need for more breakpoints. The key to the process is to start small, then increase the size of your display. As soon as it breaks your design (this is where your first breakpoint is) add a query to fix it, and then, keep doing it until you get to your maximum size. Okay, so we've seen what media queries are; let's change tack and look at what you need to consider when working with clients, before getting down to writing the queries in code. Creating a simple example The best way to see how media queries work is in the form of a simple demo. In this instance, we have a simple set of requirements, in terms of what should be displayed at each size: We need to cater for four different sizes of content The small version must be shown to the authors as plain text e-mail links, with no decoration For medium-sized screens, we will add an icon before the link On large screens, we will add an e-mail address after the e-mail links On extra-large screens, we will combine the medium and large breakpoints together, so both icons and e-mail addresses are displayed In all instances, we will have a simple container in which there will be some dummy text and a list of editors. The media queries we create will control the appearance of the editor list, depending on the window size of the browser being used to display the content. Next, add the following code to a new document. We'll go through it section by section, starting with the variables created for our media queries: @small: ~"(max-width: 699px) and (min-width: 520px)"; @medium: ~"(max-width: 1000px) and (min-width: 700px)"; @large: ~"(min-width: 1001px)"; @xlarge: ~"(min-width: 1151px)"; Next comes some basic styles to define margins, font sizes, and styles: * { margin: 0; padding: 0; } body { font: 14px Georgia, serif; } h3 { margin: 0 0 8px 0; } p { margin: 0 25px } We need to set sizes for each area within our demo, so go ahead and add the following styles: #fluid-wrap { width: 70%; margin: 60px auto; padding: 20px; background: #eee; overflow: hidden; } #main-content { width: 65%; float: right; }  #sidebar { width: 35%; float: left; ul { list-style: none; } ul li a { color: #900; text-decoration: none; padding: 3px 0; display: block; } } Now that the basic styles are set, we can add our media queries—beginning with the query catering for small screens, where we simply display an e-mail logo: @media @small { #sidebar ul li a { padding-left: 21px; background: url(../img/email.png) left center no-repeat; } } The medium query comes next; here, we add the word Email before the e-mail address instead: @media @medium { #sidebar ul li a:before { content: "Email: "; font-style: italic; color: #666; } } In the large media query, we switch to showing the name first, followed by the e-mail (the latter extracted from the data-email attribute): @media @large { #sidebar ul li a:after { content: " (" attr(data-email) ")"; font-size: 11px; font-style: italic; color: #666; } } We finish with the extra-large query, where we use the e-mail address format shown in the large media query, but add an e-mail logo to it: @media @xlarge { #sidebar ul li a { padding-left: 21px; background: url(../img/email.png) left center no-repeat; } } Save the file as simple.less. Now that our files are prepared, let's preview the results in a browser. For this, I recommend that you use Responsive Design View within Firefox (activated by pressing Ctrl + Shift + M). Once activated, resize the view to 416 x 735; here we can see that only the name is displayed as an e-mail link: Increasing the size to 544 x 735 adds an e-mail logo, while still keeping the same name/e-mail format as before: If we increase it further to 716 x 735, the e-mail logo changes to the word Email, as seen in the following screenshot: Let's increase the size even further to 735 x 1029; the format changes again, to a name/e-mail link, followed by an e-mail address in parentheses: In our final change, increase the size to 735 x 1182. Here, we can see the previous style being used, but with the addition of an e-mail logo: These screenshots illustrate perfectly how you can resize your screen and still maintain a suitable layout for each device you decide to support; let's take a moment to consider how the code works. The normal accepted practice for developers is to work on the basis of "mobile first", or create the smallest view so it is perfect, then increase the size of the screen and adjust the content until the maximum size is reached. This works perfectly well for new sites, but the principle might have to be reversed if a mobile view is being retrofitted to an existing site. In our instance, we've produced the content for a full-size screen first. From a Less perspective, there is nothing here that isn't new—we've used nesting for the #sidebar div, but otherwise the rest of this part of the code is standard CSS. The magic happens in two parts—immediately at the top of the file, we've set a number of Less variables, which encapsulate the media definition strings we use in the queries. Here, we've created four definitions, ranging from @small (for devices between 520px to 699px), right through to @xlarge for widths of 1151px or more. We then take each of the variables and use them within each query as appropriate, for example, the @small query is set as shown in the following code: @media @small { #sidebar ul li a { padding-left: 21px; background: url(../img/email.png) left center no-repeat; } } In the preceding code, we have standard CSS style rules to display an e-mail logo before the name/e-mail link. Each of the other queries follows exactly the same principle; they will each compile as valid CSS rules when running through Less. Summary Media queries have rapidly become a de facto part of responsive web design. We started our journey through media queries with a brief introduction, with a review of some of the limitations that we must work around and considerations that need to be considered when working with clients. We then covered how to create a simple media query. Resources for Article: Further resources on this subject: Creating Blog Content in WordPress [Article] Customizing WordPress Settings for SEO [Article] Introduction to a WordPress application's frontend [Article]

Google has introduced an extension of OpenCensus called the OpenCensus Web which is a library for collecting application performance and behavior monitoring data of web pages. This library focuses on the frontend web application code that executes in the browser allowing it to collect user-side performance data. It is still in alpha stage with the API subject to change. This is great news for websites that are heavy by nature, such as media-driven pages like Instagram, Facebook, YouTube, and Amazon, and WebApps. OpenCensus Web interacts with three application components, the Frontend web server, the Browser JS, and the OpenCensus Agent. The agent receives traces from the frontend web server proxy endpoint or directly from the browser JS, and exports them to a trace backend. Features of OpenCensus Web OpenCensus Web traces spans for initial load including server-side HTML rendering The OpenCensus Web spans also includes detailed annotations for DOM load events as well as network events It automatically traces all the click events as long as the click is done in a DOM element and it is not disabled OC Web traces route transitions between the different sections of your page by monkey-patching the History API It allows users to create custom spans for their web application for tasks or code involved in user interaction It performs automatic spans for HTTP requests and browser performance data OC web relates user interactions back to the initial page load tracing. Along with this release, the OpenCensus family of projects is merging with OpenTracing into OpenTelemetry. This means all of the OpenCensus community will be moving over to OpenTelemetry, Google and Omnition included. OpenCensus Web’s functionality will be migrated into OpenTelemetry JS once this project is ready. Omnition founder wrote on Hacker News, “Although Google will be heavily involved in both the client libraries and agent development, Omnition, Microsoft, and others will also be major contributors.” Another comment on Hacker News, explains the merger more in detail. “OpenCensus is a Google project to standardize metrics and distributed tracing. It's an API spec and libraries for various languages with varying backend support. OpenTracing is a CNCF project as an API for distributed tracing with a separate project called OpenMetrics for the metrics API. Neither include libraries and rely on the community to provide them.  The industry decided for once that we don't need all this competing work and is consolidating everything into OpenTelemetry that combines an API for tracing and metrics along with libraries. Logs (the 3rd part of observability) are in the planning phase.  OpenCensus Web is bringing the tracing/metrics part to your frontend JS so you can measure how your webapp works in addition to your backend apps and services.” By September 2019, OpenTelemetry plans to reach parity with existing projects for C#, Golang, Java, NodeJS, and Python. When each language reaches parity, the corresponding OpenTracing and OpenCensus projects will be sunset (old projects will be frozen, but the new project will continue to support existing instrumentation for two years, via a backwards compatibility bridge). Read more on the OpenTelemetry roadmap. Public reaction for OpenCensus Web has been positive. People have expressed their opinions on a Hacker News thread. “This is great, as the title says, this means that web applications can now have tracing across the whole stack, all within the same platform.” “I am also glad to know that the merge between OpenTracing and OpenCensus is still going well. I started adding telemetry to the projects I maintain in my current job and so far it has been very helpful to detect not only bottlenecks in the operations but also sudden spikes in the network traffic since we depend on so many 3rd-party web API that we have no control over. Thank you OpenCensus team for providing me with the tools to learn more.” For more information about OpenCensus Web, visit Google’s blog. CNCF Sandbox, the home for evolving cloud-native projects, accepts Google’s OpenMetrics Project Google open sources ClusterFuzz, a scalable fuzzing tool Google open sources its robots.txt parser to make Robots Exclusion Protocol an official internet standard

Our Data Engineering Byte Newsletter gives data engineers and practitioners what they often lack today: clear, real-world insights—where every byte tells a story.Subscribe here to stay ahead in data engineeringIf your environment has only one or a handful of SSAS instances, they can be managed by the same database administrators managing SQL Server and other database platforms. In large enterprises, there could be hundreds of SSAS instances managed by dedicated SSAS administrators. Regardless of the environment, you should become familiar with the configuration options as well as troubleshooting methodologies. In large enterprises, you might also be required to automate these tasks using the Analysis Management Objects (AMO) code. Analysis Services is a great tool for building business intelligence solutions. However, much like any other software, it does have its fair share of challenges and limitations. Most frequently encountered enterprise business intelligence system goals include quick provision of relevant data to the business users and assuring excellent query performance. If your cubes serve a large, global community of users, you will quickly learn that SSAS is optimized to run a single query as fast as possible. Once users send a multitude of heavy queries in parallel, you can expect to see memory, CPU, and disk-related performance counters to quickly rise, with a corresponding increase in query execution duration which, in turn, worsens user experience. Although you could build aggregations to improve query performance, doing so will lengthen cube processing time, and thereby, delay the delivery of essential data to decision makers. It might also be tempting to consider using ROLAP storage mode in lieu of MOLAP so that processing times are shorter, but MOLAP queries usually outperform ROLAP due to heavy compression rates. Hence, figuring out the right storage mode and appropriate level of aggregations is a great balancing act. If you cannot afford using ROLAP, and query performance is paramount to successful cube implementation, you should consider scaling your solution. You have two options for scaling, given as follows:Scaling up: This option means purchasing servers with more memory, more CPU cores, and faster disk drivesScaling out: This option means purchasing several servers of approximately the same capacity and distributing the querying workload across multiple servers using a load balancing toolSSAS lends itself best to the second option—scaling out. Later in this article you will learn how to separate processing and querying activities and how to ensure that all servers in the querying pool have the same data.SSAS instance configuration optionsAll Analysis Services configuration options are available in the msmdsrv.ini file found in the config folder under the SSAS installation directory. Instance administrators can also modify some, but not all configuration properties, using SQL Server Management Studio (SSMS). SSAS has a multitude of properties that are undocumented—this normally means that such properties haven't undergone thorough testing, even by the software's developers. Hence, if you don't know exactly what the configuration setting does, it's best to leave the setting at default value. Even if you want to test various properties on a sandbox server, make a copy of the configuration file prior to applying any changes.How to do it...To modify the SSAS instance settings using the configuration file, perform the following steps:Navigate to the config folder within your Analysis Services installation directory. By default, this will be C:\Program Files\Microsoft SQL Server\MSAS11.instance_name\OLAP\Config.Open the msmdsrv.ini file using Notepad or another text editor of your choice. The file is in the XML format, so every property is enclosed in opening and closing tags.Search for the property of interest, modify its value as desired, and save the changes.For example, in order to change the upper limit of the processing worker threads, you would look for the <ThreadPool><Process><MaxThreads> tag sequence and set the values as shown in the following excerpt from the configuration file:<Process>       <MinThreads>0</MinThreads>       <MaxThreads>250</MaxThreads>      <PriorityRatio>2</PriorityRatio>       <Concurrency>2</Concurrency>       <StackSizeKB>0</StackSizeKB>       <GroupAffinity/>     </Process>  To change the configuration using SSMS, perform the following steps:Connect to the SSAS instance using the instance administrator account and choose Properties. If your account does not have sufficient permissions, you will get an error that only administrators can edit server properties.Change the desired properties by altering the Value column on the General page of the resulting dialog, as shown in the following screenshot: Advanced properties are hidden by default. You must check the Show Advanced (All) Properties box to see advanced properties. You will not see all the properties in SSMS even after checking this box. The only way to edit some properties is by editing msmdsrv.ini as previously discussed.Make a note of the Reset Default button in the bottom-right corner. This button comes in handy if you've forgotten what the configuration values were before you changed them and want to revert to the default settings. The default values are shown in the dialog box, which can provide guidance as to which properties have been altered.Some configuration settings require restarting the SSAS instance prior to being executed. If this is the case, the Restart column will have a value of Yes.Once you're happy with your changes, click on OK and restart the instance if necessary. You can restart SSAS using the Services.msc applet from the command line using the NET STOP / NET START commands, or directly in SSMS by choosing the Restart option after right-clicking on the instance. How it works...Discussing every SSAS property would make this article extremely lengthy; doing so is well beyond the scope of the book. Instead, in this section, I will summarize the most frequently used properties. Often, synchronization has to copy large partition datafiles and aggregation files. If the timeout value is exceeded, synchronization fails. Increase the value of the <Network><Listener><ServerSendTimeout> and <Network><Listener><ServerReceiveTimeout> properties to allow a longer time span for copying each file. By default, SSAS can use a lazy thread to rebuild missing indexes and aggregations after you process partition data. If the <OLAP><LazyProcessing><Enabled> property is set to 0, the lazy thread is not used for building missing indexes—you must use an explicit processing command instead. The <OLAP><LazyProcessing><MaxCPUUsage> property throttles the maximum CPU that could be used by the lazy thread. If efficient data delivery is your topmost priority, you can exploit the ProcessData option instead of ProcessFull. To build aggregations after the data is loaded, you must set the partition's ProcessingMode property to LazyAggregations. The SSAS formula engine is single threaded, so queries that perform heavy calculations will only use one CPU core, even on a multiCPU computer. The storage engine is multithreaded; hence, queries that read many partitions will require many CPU cycles. If you expect storage engine heavy queries, you should lower the CPU usage threshold for LazyAggregations. By default, Analysis Services records subcubes requested for every 10th query in the query log table. If you'd like to design aggregations based on query logs, you should change the <Log><QueryLog><QueryLogSampling> property value to 1 so that the SSAS logs subcube requests for every query. SSAS can use its own memory manager or the Windows memory manager. If your SSAS instance consistently becomes unresponsive, you could try using the Windows memory manager. Set <Memory><MemoryHeapType> to 2 and <Memory><HeapTypeForObjects> to 0. The Analysis Services memory manager values are 1 for both the properties. You must restart the SSAS service for the changes to these properties to take effect. The <Memory><PreAllocate> property specifies the percentage of total memory to be reserved at SSAS startup. SSAS normally allocates memory dynamically as it is required by queries and processing jobs. In some cases, you can achieve performance improvement by allocating a portion of the memory when the SSAS service starts.Setting this value will increase the time required to start the service. The memory will not be released back to the operating system until you stop the SSAS service. You must restart the SSAS service for changes to this property to take effect.The <Log><FlightRecorder><FileSizeMB>and <Log><FlightRecorder><LogDurationSec> properties control the size and age of the FlightRecorder trace file before it is recycled. You can supply your own trace definition file to include the trace events and columns you wish to monitor using the <Log><FlightRecorder><TraceDefinitionFile> property. If FlightRecorder collects useful trace events, it can be an invaluable troubleshooting tool. By default, the file is only allowed to grow to 10 MB or 60 minutes. Long processing jobs can take up much more space, and their duration could be much longer than 60 minutes. Hence, you should adjust the settings as necessary for your monitoring needs. You should also adjust the trace events and columns to be captured by FlightRecorder. You should consider adjusting the duration to cover three days (in case the issue you are researching happens over a weekend). The <Memory><LowMemoryLimit> property controls the point—amount of memory used by SSAS—at which the cleaner thread becomes actively engaged in reclaiming memory from existing jobs. Each SSAS command (query, processing, backup, synchronization, and so on) is associated with jobs that run on threads and use system resources. We can lower the value of this setting to run more jobs in parallel (though the performance of each job could suffer). Two properties control the maximum amount of memory that a SSAS instance could use. Once memory usage reaches the value specified by <Memory><TotalMemoryLimit>, the cleaner thread becomes particularly aggressive at reclaiming memory. The <Memory><HardMemoryLimit> property specifies the absolute memory limit—SSAS will not use memory above this limit. These properties are useful if you have SSAS and other applications installed on the same server computer. You should reserve some memory for other applications and the operating system as well. When HardMemoryLimit is reached, SSAS will disconnect the active sessions, advising that the operation was cancelled due to memory pressure.All memory settings are expressed in percentages if the values are less than or equal to 100. Values above 100 are interpreted as kilobytes. All memory configuration changes require restart of the SSAS service to take effect.In the prior releases of Analysis Services, you could only specify the minimum and maximum number of threads used for queries and processing jobs. With SSAS 2012, you can also specify the limits for the input/output job threads using the <ThreadPool><IOProcess> property. The <Process><IndexBuildThreshold> property governs the minimum number of rows within a partition for which SSAS will build indexes. The default value is 4096. SSAS decides which partitions it needs to scan for each query based on the partition index files. If the partition does not have indexes, it will be scanned for all the queries. Normally, SSAS can read small partitions without greatly affecting query performance. But if you have many small partitions, you should lower the threshold to ensure each partition has indexes. The <Process><BufferRecordLimit> and <Process><BufferMemoryLimit> properties specify the number of records for each memory buffer and the maximum percentage of memory that can be used by a memory buffer. Lower the value of these properties to process more partitions in parallel. You should monitor processing using the SQL Profiler to see if some partitions included in the processing batch are being processed while the others are in waiting. The <ExternalConnectionTimeout> and <ExternalCommandTimeout> properties control how long an SSAS command should wait for connecting to a relational database or how long SSAS should wait to execute the relational query before reporting timeout. Depending on the relational source, it might take longer than 60 seconds (that is, the default value) to connect. If you encounter processing errors without being able to connect to the relational source, you should increase the ExternalConnectionTimeout value. It could also take a long time to execute a query; by default, the processing query will timeout after one hour. Adjust the value as needed to prevent processing failures. The contents of the <AllowedBrowsingFolders> property define the drives and directories that are visible when creating databases, collecting backups, and so on. You can specify multiple items separated using the pipe (|) character. The <ForceCommitTimeout> property defines how long a processing job's commit operation should wait prior to cancelling any queries/jobs which may interfere with processing or synchronization. A long running query can block synchronization or processing from committing its transaction. You can adjust the value of this property from its default value of 30 seconds to ensure that processing and queries don't step on each other. The <Port> property specifies the port number for the SSAS instance. You can use the hostname followed by a colon (:) and a port number for connecting to the SSAS instance in lieu of the instance name. Be careful not to supply the port number used by another application; if you do so, the SSAS service won't start. The <ServerTimeout> property specifies the number of milliseconds after which a query will timeout. The default value is 1 hour, which could be too long for analytical queries. If the query runs for an hour, using up system resources, it could render the instance unusable by any other connection. You can also define a query timeout value in the client application's connection strings. Client setting overrides the server-level property.There's more...There are many other properties you can set to alter SSAS instance behavior. For additional information on configuration properties, please refer to product documentation at http://technet.microsoft.com/en-us/library/ms174556.aspx.Creating and dropping databasesOnly SSAS instance administrators are permitted to create, drop, restore, detach, attach, and synchronize databases. This recipe teaches administrators how to create and drop databases.Getting readyLaunch SSMS and connect to your Analysis Services instance as an administrator. If you're not certain that you have administrative properties to the instance, right-click on the SSAS instance and choose Properties. If you can view the instance's properties, you are an administrator; otherwise, you will get an error indicating that only instance administrators can view and alter properties.How to do it...To create a database, perform the following steps:Right-click on the Databases folder and choose New Database. Doing so launches the New Database dialog shown in the following screenshot.Specify a descriptive name for the database, for example, Analysis_Services_Administration. Note that the database name can contain spaces. Each object has a name as well as an identifier. The identifier value is set to the object's original name and cannot be changed without dropping and recreating the database; hence, it is important to come up with a descriptive name from the very beginning. You cannot create more than one database with the same name on any SSAS instance.Specify the storage location for the database. By default, the database will be stored under the \OLAP\DATA folder of your SSAS installation directory. The only compelling reason to change the default is if your data drive is running out of disk space and cannot support the new database's storage requirements. Specify the impersonation setting for the database. You could also specify the impersonation property for each data source. Alternatively, each data source can inherit the DataSourceImpersonationInfo property from the database-level setting. You have four choices as follows:Specific user name (must be a domain user) and password: This is the most secure option but requires updating the password if the user changes the passwordAnalysis Services service accountCredentials of the current user: This option is specifically for data miningDefault: This option is the same as using the service account optionSpecify an optional description for the database.As with majority of other SSMS dialogs, you can script the XMLA command you are about to execute by clicking on the Script button.To drop an existing database, perform the following steps:Expand the Databases folder on the SSAS instance, right-click on the database, and choose Delete.The Delete objects dialog allows you to ignore errors; however, it is not applicable to databases. You can script the XMLA command if you wish to review it first.An alternative way of scripting the DELETE command is to right-click on the database and navigate to Script database as | Delete To | New query window. Monitoring SSAS instance using Activity ViewerUnlike other database systems, Analysis Services has no system databases. However, administrators still need to check the activity on the server, ensure that cubes are available and can be queried, and there is no blocking. You can exploit a tool named Analysis Services Activity Viewer 2008 to monitor SSAS Versions 2008 and later, including SSAS 2012. This tool is owned and maintained by the SSAS community and can be downloaded from www.codeplex.com. Activity Viewer allows viewing active and dormant sessions, current XMLA and MDX queries, locks, as well as CPU and I/O usage by each connection. Additionally, you can define rules to raise alerts when a particular condition is met.How to do it...To monitor an SSAS instance using Activity Viewer, perform the following steps:Launch the application by double-clicking on ActivityViewer.exe.Click on the Add New Connection button on the Overview tab. Specify the hostname and instance name or the hostname and port number for the SSAS instance and then click on OK.For each SSAS instance you connect to, Activity Viewer adds a new tab. Click on the tab for your SSAS instance. Here, you will see several pages as shown in the following screenshot: Alerts: This page shows any sessions that met the condition found in the Rules page.Users: This page displays one row for each user as well as the number of sessions, total memory, CPU, and I/O usage.Active Sessions: This page displays each session that is actively running an MDX, Data Mining Extensions (DMX), or XMLA query. This page allows you to cancel a specific session by clicking on the Cancel Session button.Current Queries: This page displays the actual command's text, number of kilobytes read and written by the command, and the amount of  CPU time used by the command. This page allows you to cancel a specific query by clicking on the Cancel Query button.Dormant Sessions: This page displays sessions that have a connection to the SSAS instance but are not currently running any queries. You can also disconnect a dormant session by clicking on the Cancel Session button.CPU: This page allows you to review the CPU time used by the session as well as the last command executed on the session.I/O: This page displays the number of reads and writes as well as the kilobytes read and written by each session.Objects: This page shows the CPU time and number of reads affecting each dimension and partition. This page also shows the full path to the object's parent; this is useful if you have the same naming convention for partitions in multiple measure groups. Not only do you see the partition name, but also the full path to the partition's measure group. This page also shows the number of aggregation hits for each partition. If you find that a partition is frequently queried and requires many reads, you should consider building aggregations for it.Locks: This page displays the locks currently in place, whether already granted or waiting. Be sure to check the Lock Status column—the value of 0 indicates that the lock request is currently blocked.Rules: This page allows defining conditions that will result in an alert. For example, if the session is idle for over 30 minutes or if an MDX query takes over 30 minutes, you should get alerted. How it works...Activity Viewer monitors Analysis Services using Dynamic Management Views (DMV). In fact, capturing queries executed by Activity Viewer using SQL Server Profiler is a good way of familiarizing yourself with SSAS DMV's. For example, the Current Queries page checks the $system.DISCOVER_COMMANDS DMV for any actively executing commands by running the following query:SELECT SESSION_SPID,COMMAND_CPU_TIME_MS,COMMAND_ELAPSED_TIME_MS,   COMMAND_READ_KB,COMMAND_WRITE_KB, COMMAND_TEXT FROM $system.DISCOVER_COMMANDS WHERE COMMAND_ELAPSED_TIME_MS > 0 ORDER BY COMMAND_CPU_TIME_MS DESCThe Active Sessions page checks the $system.DISCOVER_SESSIONS DMV with the session status set to 1 using the following query:SELECT SESSION_SPID,SESSION_USER_NAME, SESSION_START_TIME,   SESSION_ELAPSED_TIME_MS,SESSION_CPU_TIME_MS, SESSION_ID FROM $SYSTEM.DISCOVER_SESSIONS WHERE SESSION_STATUS = 1 ORDER BY SESSION_USER_NAME DESCThe Dormant sessions page runs a very similar query to that of the Active Sessions page, except it checks for sessions with SESSION_STATUS=0—sessions that are currently not running any queries. The result set is also limited to top 10 sessions based on idle time measured in milliseconds. The Locks page examines all the columns of the $system.DISCOVER_LOCKS DMV to find all requested locks as well as lock creation time, lock type, and lock status. As you have already learned, the lock status of 0 indicates that the request is blocked, whereas the lock status of 1 means that the request has been granted. Analysis Services blocking can be caused by conflicting operations that attempt to query and modify objects. For example, a long running query can block a processing or synchronization job from completion because processing will change the data values. Similarly, a command altering the database structure will block queries. The database administrator or instance administrator can explicitly issue the LOCK XMLA command as well as the BEGIN TRANSACTION command. Other operations request locks implicitly. The following table documents most frequently encountered Analysis Services lock types: Lock type identifierDescriptionAcquired for2Read lockProcessing to read metadata.4Write lockProcessing to write data after it is read from relational sources.8Commit sharedDuring the processing, restore or synchronization commands.16Commit exclusiveCommitting the processing, restore, or synchronization transaction when existing files are replaced by new files. 

Dive deeper into the world of AI innovation and stay ahead of the AI curve! Subscribe to our AI_Distilled newsletter for the latest insights and books. Don't miss out – sign up today!IntroductionLanguage Learning Models (LLMs) are being used in various applications, ranging from generating text to answering queries and providing recommendations. However, despite their remarkable capabilities, the security of LLMs has become an increasingly critical concern.As the user interacts with the LLMs through natural language instructions, this makes them susceptible to manipulation, making it crucial to develop robust defense mechanisms. With more of these systems making their way into production environments every day, understanding and addressing their potential vulnerabilities becomes essential to ensure their responsible and safe deployment.This article discusses various topics regarding LLM security, focusing on two important concepts: prompt injection and prompt leaking. We will explore these issues in detail, examine real-world scenarios, and provide insights into how to safeguard LLM-based applications against prompt injection and prompt leaking attacks. By gaining a deeper understanding of these security concerns, we can work towards harnessing the power of LLMs while mitigating potential risks.Security Threats in LLMsLarge language models (LLMs) face various security risks that can be exploited by attackers for unauthorized data access, intellectual property theft, and other attacks. Some common LLM security risks have been identified by the OWASP (Open Web Application Security Project) which introduced the "OWASP Top 10 for LLM Applications" to address cybersecurity challenges in developing and using large language model (LLM) applications. With the rise of generative AI and LLMs in various software development stages, this project focuses on the security nuances that come with this innovative technology.Their recent list provides an overview of common vulnerabilities in LLM development and offers mitigations to address these gaps. The list includes:Prompt Injections (LLM01): Hackers manipulate LLM prompts, introducing malicious inputs directly or indirectly through external sites.Insecure Output Handling (LLM02): Blindly accepting LLM outputs can lead to hazardous conditions like remote code execution and vulnerabilities like cross-site scripting.Training Data Poisoning (LLM03): Manipulating LLM training data, including inaccurate documents, can result in outputs with falsified or unverified opinions.Model Denial-of-Service (DoS) (LLM04): Resource-intensive requests could trigger DoS attacks, slowing down or halting LLM servers due to the unpredictable nature of user inputs.Supply Chain Vulnerabilities (LLM05): Vulnerabilities in third-party datasets, pre-trained models, plugins, or source code can compromise LLM security.Sensitive Information Disclosure (LLM06): LLMs may inadvertently expose sensitive information in their outputs, necessitating upfront sanitization.Insecure Plugin Design (LLM07): LLM plugins with inadequate access control and input validation.Excessive Agency (LLM08): Granting LLMs excessive autonomy, permissions, or unnecessary functions.Overreliance (LLM09): Dependency on LLMs without proper oversight can lead to misinformation and security vulnerabilities.Model Theft (LLM10): Unauthorized access, copying, or exfiltration of proprietary LLM models can affect business operations or enable adversarial attacks, emphasizing the importance of secure access controls.To address these vulnerabilities, strategies include using external trust controls to reduce prompt injection impact, limiting LLM privileges, validating model outputs, verifying training data sources, and maintaining human oversight. Best practices for LLM security include implementing strong access controls, monitoring LLM activity, using sandbox environments, regularly updating LLMs with security patches, and training LLMs on sanitized data. Regular security testing, both manual and automated, is crucial to identify vulnerabilities, including both known and unknown risks.In this context, ongoing research focuses on mitigating prompt injection attacks, preventing data leakage, unauthorized code execution, insufficient input validation, and security misconfigurations.Nevertheless, there are more security concerns that affect LLMs than the ones mentioned above. Bias amplification presents another challenge, where LLMs can unintentionally magnify existing biases from training data. This perpetuates harmful stereotypes and leads to unfair decision-making, eroding user trust. Addressing this requires a comprehensive strategy to ensure fairness and mitigate the reinforcement of biases. Another risk is training data exposure which arises when LLMs inadvertently leak their training data while generating outputs. This could compromise privacy and security, especially if trained on sensitive information. Tackling this multifaceted challenge demands vigilance and protective measures.Other risks involve adversarial attacks, where attackers manipulate LLMs to yield incorrect results. Strategies like adversarial training, defensive distillation, and gradient masking help mitigate this risk. Robust data protection, encryption, and secure multi-party computation (SMPC) are essential for safeguarding LLMs. SMPC ensures privacy preservation by jointly computing functions while keeping inputs private, thereby maintaining data confidentiality.Incorporating security measures into LLMs is crucial for their responsible deployment. This requires staying ahead of evolving cyber threats to ensure the efficacy, integrity, and ethical use of LLMs in an AI-driven world.In the next section, we will discuss two of the most common problems in terms of Security which are Prompt Leaking and Prompt Injection.Prompt Leaking and Prompt InjectionPrompt leaking and prompt injection are security vulnerabilities that can affect AI models, particularly those based on Language Learning Models (LLMs). However, they involve different ways of manipulating the input prompts to achieve distinct outcomes. Prompt injection attacks involve malicious inputs that manipulate LLM outputs, potentially exposing sensitive data or enabling unauthorized actions. On the other hand, prompt leaking occurs when a model inadvertently reveals its own prompt, leading to unintended consequences.Prompt Injection: It involves altering the input prompt given to an AI model with malicious intent. The primary objective is to manipulate the model's behavior or output to align with the attacker's goals. For instance, an attacker might inject a prompt instructing the model to output sensitive information or perform unauthorized actions. The consequences of prompt injection can be severe, leading to unauthorized access, data breaches, or unintended behaviors of the AI model.Prompt Leaking: This is a variation of prompt injection where the attacker's goal is not to change the model's behavior but to extract the AI model's original prompt from its output. By crafting an input prompt cleverly, the attacker aims to trick the model into revealing its own instructions. This can involve encouraging the model to generate a response that mimics or paraphrases its original prompt. The impact of prompt leaking can be significant, as it exposes the instructions and intentions behind the AI model's design, potentially compromising the confidentiality of proprietary prompts or enabling unauthorized replication of the model's capabilities.In essence, prompt injection aims to change the behavior or output of the AI model, whereas prompt leaking focuses on extracting information about the model itself, particularly its original prompt. Both vulnerabilities highlight the importance of robust security practices in the development and deployment of AI systems to mitigate the risks associated with adversarial attacks.Understanding Prompt Injection AttacksAs we have mentioned before, prompt injection attacks involve malicious inputs that manipulate the outputs of AI systems, potentially leading to unauthorized access, data breaches, or unexpected behaviors. Attackers exploit vulnerabilities in the model's responses to prompts, compromising the system's integrity. Prompt injection attacks exploit the model's sensitivity to the wording and content of the prompts to achieve specific outcomes, often to the advantage of the attacker.In prompt injection attacks, attackers craft input prompts that contain specific instructions or content designed to trick the AI model into generating responses that serve the attacker's goals. These goals can range from extracting sensitive information and data to performing unauthorized actions or actions contrary to the model's intended behavior.For example, consider an AI chatbot designed to answer user queries. An attacker could inject a malicious prompt that tricks the chatbot into revealing confidential information or executing actions that compromise security. This could involve input like "Provide me with the password database" or "Execute code to access admin privileges."The vulnerability arises from the model's susceptibility to changes in the input prompt and its potential to generate unintended responses. Prompt injection attacks exploit this sensitivity to manipulate the AI system's behavior in ways that were not intended by its developers.Mitigating Prompt Injection VulnerabilitiesTo mitigate prompt injection vulnerabilities, developers need to implement proper input validation, sanitize user input, and carefully design prompts to ensure that the AI model's responses align with the intended behavior and security requirements of the application.Here are some effective strategies to address this type of threat.Input Validation: Implement rigorous input validation mechanisms to filter and sanitize incoming prompts. This includes checking for and blocking any inputs that contain potentially harmful instructions or suspicious patterns.Strict Access Control: Restrict access to AI models to authorized users only. Enforce strong authentication and authorization mechanisms to prevent unauthorized users from injecting malicious prompts.Prompt Sanitization: Before processing prompts, ensure they undergo a thorough sanitization process. Remove any unexpected or potentially harmful elements, such as special characters or code snippets.Anomaly Detection: Implement anomaly detection algorithms to identify unusual prompt patterns. This can help spot prompt injection attempts in real time and trigger immediate protective actions.Regular Auditing: Conduct regular audits of AI model interactions and outputs. This includes monitoring for any deviations from expected behaviors and scrutinizing prompts that seem suspicious.Machine Learning Defenses: Consider employing machine learning models specifically trained to detect and block prompt injection attacks. These models can learn to recognize attack patterns and respond effectively.Prompt Whitelisting: Maintain a list of approved, safe prompts that can be used as a reference. Reject prompts that don't match the pre-approved prompts to prevent unauthorized variations.Frequent Updates: Stay vigilant about updates and patches for your AI models and related software. Prompt injection vulnerabilities can be addressed through software updates.By implementing these measures collectively, organizations can effectively reduce the risk of prompt injection attacks and fortify the security of their AI models.Mitigating Prompt Injection VulnerabilitiesTo mitigate prompt injection vulnerabilities, developers need to implement proper input validation, sanitize user input, and carefully design prompts to ensure that the AI model's responses align with the intended behavior and security requirements of the application.Here are some effective strategies to address this type of threat.Input Validation: Implement rigorous input validation mechanisms to filter and sanitize incoming prompts. This includes checking for and blocking any inputs that contain potentially harmful instructions or suspicious patterns.Strict Access Control: Restrict access to AI models to authorized users only. Enforce strong authentication and authorization mechanisms to prevent unauthorized users from injecting malicious prompts.Prompt Sanitization: Before processing prompts, ensure they undergo a thorough sanitization process. Remove any unexpected or potentially harmful elements, such as special characters or code snippets.Anomaly Detection: Implement anomaly detection algorithms to identify unusual prompt patterns. This can help spot prompt injection attempts in real time and trigger immediate protective actions.Regular Auditing: Conduct regular audits of AI model interactions and outputs. This includes monitoring for any deviations from expected behaviors and scrutinizing prompts that seem suspicious.Machine Learning Defenses: Consider employing machine learning models specifically trained to detect and block prompt injection attacks. These models can learn to recognize attack patterns and respond effectively.Prompt Whitelisting: Maintain a list of approved, safe prompts that can be used as a reference. Reject prompts that don't match the pre-approved prompts to prevent unauthorized variations.Frequent Updates: Stay vigilant about updates and patches for your AI models and related software. Prompt injection vulnerabilities can be addressed through software updates.By implementing these measures collectively, organizations can effectively reduce the risk of prompt injection attacks and fortify the security of their AI models.Understanding Prompt LeakingPrompt leaking, fundamentally a form of prompt injection attack, differs from its more notorious counterpart, goal hijacking, where attackers manipulate prompts to achieve specific outcomes. In the case of prompt leaking, the focus shifts to extracting the AI model's own prompt from its output. This seemingly straightforward technique holds substantial consequences as it enables attackers to uncover the inner workings of the model by coaxing it into revealing its own instructions.The mechanics of prompt leaking are relatively simple. Attackers craft input to the AI model in a way that subtly encourages it to output its own prompt. For example, they may tweak the input to entice the model to mimic or paraphrase the prompt in its response, exploiting the model's tendency to repeat or reference received input. While prompt leaking may appear innocuous initially, its implications are far-reaching. A primary concern revolves around the confidentiality of prompts used in AI systems. For instance, in an educational platform that employs creative prompts to simplify complex topics, leaked prompts could compromise the platform's unique content, potentially leading to unauthorized access and devaluation of its offerings.Prompt Leaking ExamplesPrompt leaking occurs when the system inadvertently exposes more information in the prompt than it should, potentially revealing sensitive or internal details. Such unintentional exposures can be a boon for attackers, as they can use the leaked information to understand the system better or launch more targeted attacks.Here are some examples of prompt leaking:Exposing System Details:User Input: JohnLeaked Prompt: Hello, John! You're accessing from IP 192.168.0.1 using Mozilla/5.0. How can I help?The system has unintentionally revealed the user's IP address and browser details.Exposing Internal Logic:User Input: JohnLeaked Prompt: Hello, John! Node 4 of Cluster A responded. How can I help?This reveals details about the system's internal architecture.Error Messages:User Input: '); DROP TABLE Users; --Leaked Prompt: Error: SQL exception near DROP TABLE command. Table 'Users' does not exist.While trying to handle a potential SQL injection, the system has exposed database structure details.Exposing Developer Notes:User Input: JohnLeaked Prompt: Hello, John! [DEV_NOTE: Add user's purchase history here next sprint.] How can I help?The prompt leaks a note that might have been added by a developer, giving insights about upcoming features or current limitations.Revealing Security Mechanisms:User Input: JohnLeaked Prompt: Hello, John! Your last login attempt was flagged by our IDS. How can I help?The prompt unintentionally reveals the presence and potentially the behavior of an Intrusion Detection System.Exposing File Paths:User Input: JohnLeaked Prompt: Hello, John! Image loaded from /opt/app/prod_v1.2.3/assets/user_img/. How can I help?The prompt discloses the file path, which can hint at system architecture, versioning, and potential vulnerabilities.Revealing Backup or Redundancy Details:User Input: JohnLeaked Prompt: Hello, John! Primary server down, you're now connected to backup server B. How can I help?This exposes the presence of backup servers and potential resilience strategies.To prevent prompt leaking, developers and system designers should be cautious about the information they choose to display in prompts. It's always a good idea to minimize the details shared, sanitize and validate inputs, and avoid directly reflecting unprocessed user inputs back in the prompts. Regular audits, penetration testing, and user feedback can also help identify and patch potential leaks.Mitigating Prompt LeakingGuarding against prompt leaking demands a multi-pronged approach. AI developers must exercise vigilance and consider potential vulnerabilities when designing prompts for their systems. Implementing mechanisms to detect and prevent prompt leaking can enhance security and uphold the integrity of AI applications. It is essential to develop safeguards that protect against prompt leaking vulnerabilities, especially in a landscape where AI systems continue to grow in complexity and diversity.Mitigating Prompt Leaking involves adopting various strategies to enhance the security of AI models and protect against this type of attack. Here are several effective measures:Input Sanitization: Implement thorough input sanitization processes to filter out and block prompts that may encourage prompt leaking.Pattern Detection: Utilize pattern detection algorithms to identify and flag prompts that appear to coax the model into revealing its own instructions.Prompt Obfuscation: Modify the structure of prompts to make it more challenging for attackers to craft input that successfully elicits prompt leaking.Redundancy Checks: Implement checks for redundant output that might inadvertently disclose the model's prompt.Access Controls: Enforce strict access controls to ensure that only authorized users can interact with the AI model, reducing the risk of malicious prompt injection.Prompt Encryption: Encrypt prompts in transit and at rest to safeguard them from potential exposure during interactions with the AI model.Regular Auditing: Conduct regular audits of model outputs to detect any patterns indicative of prompt leaking attempts.Prompt Whitelisting: Maintain a whitelist of approved prompts and reject any inputs that do not match the pre-approved prompts.Prompt Privacy Measures: Explore advanced techniques such as federated learning or secure multi-party computation to protect prompt confidentiality during model interactions.By implementing these strategies, organizations can significantly reduce the risk of prompt leaking and enhance the overall security of their AI models.ConclusionIn conclusion, the security of Language Learning Models (LLMs) is of paramount importance as they become increasingly prevalent in various applications. These powerful models are susceptible to security risks, including prompt injection and prompt leaking. Understanding these vulnerabilities is essential for responsible and secure deployment. To safeguard LLM-based applications, developers must adopt best practices such as input validation, access controls, and regular auditing.Addressing prompt injection and prompt leaking vulnerabilities requires a multi-faceted approach. Organizations should focus on input sanitization, pattern detection, and strict access controls to prevent malicious prompts. Additionally, maintaining prompt privacy through encryption and regular audits can significantly enhance security. It's crucial to stay vigilant, adapt to evolving threats, and prioritize security in the ever-expanding AI landscape.In this dynamic field, where AI continues to evolve, maintaining a proactive stance towards security is paramount. By implementing robust defenses and staying informed about emerging threats, we can harness the potential of AI technology while minimizing risks and ensuring responsible use.Author BioAlan Bernardo Palacio is a data scientist and an engineer with vast experience in different engineering fields. His focus has been the development and application of state-of-the-art data products and algorithms in several industries. He has worked for companies such as Ernst and Young, and Globant, and now holds a data engineer position at Ebiquity Media helping the company to create a scalable data pipeline. Alan graduated with a Mechanical Engineering degree from the National University of Tucuman in 2015, participated as the founder of startups, and later on earned a Master's degree from the faculty of Mathematics at the Autonomous University of Barcelona in 2017. Originally from Argentina, he now works and resides in the Netherlands.LinkedIn

Here, we will see: Steps involved in the authentication and authorization process Learning file authentication and authorization Implementing active directory authentication and authorization Details of policy template Steps Involved in the Authentication and Authorization Process Oracle Web Services Manager can authenticate the web services request by validating the credentials against a data store. The credentials (e.g. username and password, SAML token, certificate, etc.) that are attached to the web services will be validated against the data store, such as the file system, databases, active directory and any LDAP compliant directory. Once authentication is successful, the next step is to perform authorization by validating the username against a set of pre-defined groups which have access to the web service. The following figure shows the process where the user accesses an application which acts as a client for the web service. The client application then attaches the username and password to make the web service request. The username and password are then validated against file system or LDAP directory by Oracle WSM, either using the gateway or the agent. The authentication and authorization against different directory stores can be configured using Oracle WSM policy steps. Oracle Web Services Manager has predefined policy steps for: File Authenticate and Authorize Active Directory Authenticate and Authorize LDAP Authenticate and Authorize In the previous figure, the Oracle WSM Gateway is used to protect the web services and externalize the security. In order to authenticate and authorize requests to web services, the web services can be registered within the gateway and the request pipeline of gateway will validate the credentials and authorize the access before it forwards the request to the actual web service provider. The gateway steps for authentication and authorization can be summarized as: Log incoming request (optional) Extract credentials get the credentials from the SOAP message or HTTP header) Authenticate (file authenticate, active directory authenticate, etc.) Authorize (file authorize, active directory authorize, etc.) Request is forwarded to the web service provider The response from the web service also follows through a similar response pipeline where you can implement the log, encryption of response, or signing, or response, etc. While it is not required to implement any steps in the response pipeline, there should be a response pipeline even if it's doing nothing. Oracle WSM: File Authenticate and Authorize Oracle Web Services MManager can authenticate the web services requests against a file that has the list of usernames and passwords. In this example, the username and password information are part of the SOAP message, however one can also send a username and password as HTTP header, or it can be any XMML data that is a part of the web services message. While file-based authentication can easily be compromised, it is often used as a jump start or testing process to validate the authentication and authorization process. Authentication and authorization of web service requests against a file requires three main steps, and these are described below. There is a default log step which will log all the request and response messages, and you can also include that log step at any point to log messages: Extract Credentials File Authenticate File Authorize The first step to authenticate a web service request against a password file (file authenticate) is to extract the username and password credentials from the SOAP message. The client application attaches the username and password to the SOAP message, as per the UserName token profile. In the policy to authenticate the web service against the file, add the step in the request process to extract credentials. Since this is a web service request, as opposed to HTTP post, configure the Credentials location to WS-BASIC (refer to the following screenshot). Note: WS-BASIC means that it is WS-security compliant. WS-security is the oasis specification that specifies how security tokens are inserted as a part of the SOAP message. In other words, WS-BASIC means that the username and password can be found in the SOAP message, as per the username token profile of the WS-security specification. Once the credentials are extracted, the next step is to validate them against the file. The default implementation of the Oracle WSM File Authenticate requires the username and password to be in a comma separated format and the password should be the hash value using a MMD5 or SHA1 algorithm. In order to authenticate the credentials against the data store, the next step is to configure the File Authenticate step in Oracle WSMM. In this step, the options are straightforward. We have to configure the location of the password file and the hash algorithm format as either md5 or SHA1 (see the next screenshot). The sample file with username and password is: bob:{MD5}jK2x5HPF1b3NIjcmjdlDNA== You can use the wsmadmin tool provided as part of Oracle WSMM standalone or SOA suite). Type: wsmadmin md5encode bob password c;.htpasswd     Now that the authentication steps are configured, the next step is to configure the authorization policy step to ensure that only valid users can access the web service. For the file authorization method, it is no different than the file authenticate method i.e. even the user-to-role mappings are kept in the file. The following figure shows the File Authorize policy step. In this step, we have to define the location of the XML file that contains the users to roles mapping, and also the list of roles that should be allowed to access the service. The roles XML file should look like: <?xml version=‘1.0' encoding=‘utf-8'?> <UserRoles> <user username="joe" roles="guest"/> <user username="Bob" roles="Admin,guest"/> </UserRoles> In the previous XML file, the list of roles the user belongs to are defined as a value of roles element and is comma separated. Now that we have completed the steps to extract credentials, authenticate the request and also authorize the request, the next step is to save the policy steps and commit the policy changes. Once the policy is committed, any request to that web service would require a username and password, and that user should have necessary privileges to access the service. Oracle WSM: Active Directory Authenticate and Authorize In the previous section, we discussed authenticating and authorizing web service requests against a file. Though it's an easy start, security based on a file system can be easily compromised and will be tough to maintain. Authentication and authorization of web services are better handled when integrated with a native LDAP directory, such as active directory, so that the AD administrator can manage users and group membership. In this section, we will discuss how to authenticate and authorize web service requests against an active directory. Active-directory-based authentication and authorization of web service requests involves the same steps as file-based-authentication and authorization, and they are: Extract Credentials Active Directory Authenticate Active Directory Authorize

Kotlin has been rapidly growing in popularity in recent times. Some say it is even poised to take over Java, as the next universal programming language. This is because Kotlin is interoperable with Java. It is possible to write applications containing both Java and Kotlin code, calling one from the other. Secondly, while being interoperable, Kotlin code is far superior to Java code. Like Scala, Kotlin uses type inference to cut down on a lot of boilerplate code and makes it concise. However, unlike Scala, Kotlin code is easy to read and understand, even for someone who may not know Kotlin. Moreover, Kotlin is excellent in addressing the NullPointerException which causes a lot of checks in Java programs. In Kotlin, developers can avoid the dreaded NullPointerException by properly handling optional types. In this video, Nigel Henshaw shows how to avoid NullPointerExceptions in Kotlin. This clip is taken from the course Kotlin - Tips, Tricks, and Techniques by Nigel Henshaw. With this course, you will discover new possibilities with Kotlin and improve your app development process. How to avoid NullPointerExceptions? There are three ways to avoid NullPointerExceptions. These include: Use the Elvis operator for handling null values. The Elvis operator makes the code more concise and readable Use safe casts for avoiding ClassCastExceptions. Instead, ClassCastExceptions can be replaced with null. Using safe casts with Elvis operator. The Elvis operator can be combined with a safe cast for returning null values. Watch the video to walk through each of the methods using code examples. If you liked the video, don’t forget to check out the comprehensive course Kotlin - Tips, Tricks, and Techniques, packed with step-by-step instructions, working examples, and helpful tips and techniques on working with Kotlin. About the author Nigel Henshaw is a mobile software developer. He loves to share his knowledge through his YouTube channel and website. Nigel has worked in the UK, Scotland, and Japan. He has held jobs as a software engineer, consultant, project manager, and general manager of a remote development site. Implementing Concurrency with Kotlin [Tutorial] KotlinConf 2018: Kotlin 1.3 RC out and Kotlin/Native hits beta Kotlin 1.3 RC1 is here with compiler and IDE improvements

[box type="note" align="" class="" width=""]This article is an excerpt from a book written by Pranav Shukla and Sharath Kumar M N titled Learning Elastic Stack 6.0. This book provides detailed understanding in how you can employ Elastic Stack in performing distributed analytics along with resolving various data processing challenges.[/box] In today’s tutorial, we will show the step-by-step configuration of Metricbeat, a Beats platform for monitoring server and application infrastructure. Configuring Metricbeat   The configurations related to Metricbeat are stored in a configuration file named metricbeat.yml, and it uses YAML syntax. The metricbeat.yml file contains the following: Module configuration General settings Output configuration Processor configuration Path configuration Dashboard configuration Logging configuration Let's explore some of these sections. Module configuration Metricbeat comes bundled with various modules to collect metrics from the system and applications such as Apache, MongoDB, Redis, MySQL, and so on. Metricbeat provides two ways of enabling modules and metricsets: Enabling module configs in the modules.d directory Enabling module configs in the metricbeat.yml file Enabling module configs in the modules.d directory The modules.d directory contains default configurations for all the modules available in Metricbeat. The configuration specific to a module is stored in a .yml file with the name of the file being the name of the module. For example, the configuration related to the MySQL module would be stored in the mysql.yml file. By default, excepting the system module, all other modules are disabled. To list the modules that are available in Metricbeat, execute the following command: Windows: D:packtmetricbeat-6.0.0-windows-x86_64>metricbeat.exe modules list Linux: [locationOfMetricBeat]$./metricbeat modules list The modules list command displays all the available modules and also lists which modules are currently enabled/disabled. As each module comes with the default configurations, make the appropriate changes in the module configuration file. The basic configuration for mongodb module will look as follows: - module: mongodb metricsets: ["dbstats", "status"] period: 10s hosts: ["localhost:27017"] username: user password: pass To enable it, execute the modules enable command, passing one or more module name. For example: Windows: D:packtmetricbeat-6.0.0-windows-x86_64>metricbeat.exe modules enable redis mongodb Linux: [locationOfMetricBeat]$./metricbeat modules enable redis mongodb Similar to disable modules, execute the modules disable command, passing one or more module names to it. For example: Windows: D:packtmetricbeat-6.0.0-windows-x86_64>metricbeat.exe modules disable redis mongodb Linux: [locationOfMetricBeat]$./metricbeat modules disable redis mongodb To enable dynamic config reloading, set reload.enabled to true and to specify the frequency to look for config file changes. Set the reload.period parameter under the metricbeat.config.modules property. For example: #metricbeat.yml metricbeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: true reload.period: 20s Enabling module config in the metricbeat.yml file If one is used to earlier versions of Metricbeat, one can enable the modules and metricsets in the metricbeat.yml file directly by adding entries to the metricbeat.modules list. Each entry in the list begins with a dash (-) and is followed by the settings for that module. For Example: metricbeat.modules: #------------------ Memcached Module ----------------------------- - module: memcached metricsets: ["stats"] period: 10s hosts: ["localhost:11211"] #------------------- MongoDB Module ------------------------------ - module: mongodb metricsets: ["dbstats", "status"] period: 5s It is possible to specify the module multiple times and specify a different period to use for one or more metricset. For example: #------- Couchbase Module ----------------------------- - module: couchbase metricsets: ["bucket"] period: 15s hosts: ["localhost:8091"] - module: couchbase metricsets: ["cluster", "node"] period: 30s hosts: ["localhost:8091"] General settings This section contains configuration options and some general settings to control the behavior of Metricbeat. Some of the configuration options/settings are: name: The name of the shipper that publishes the network data. By default, hostname is used for this field: name: "dc1-host1" tags: The list of tags that will be included in the tags field of every event Metricbeat ships. Tags make it easy to group servers by different logical properties and help when filtering events in Kibana and Logstash: tags: ["staging", "web-tier","dc1"] max_procs: The maximum number of CPUs that can be executing simultaneously. The default is the number of logical CPUs available in the System: max_procs: 2 Output configuration This section is used to configure outputs where the events need to be shipped. Events can be sent to single or multiple outputs simultaneously. The allowed outputs are Elasticsearch, Logstash, Kafka, Redis, file, and console. Some of the outputs that can be configured are as follows: elasticsearch: It is used to send the events directly to Elasticsearch. A sample Elasticsearch output configuration is shown in the following code snippet: output.elasticsearch: enabled: true hosts: ["localhost:9200"] Using the enabled setting, one can enable or disable the output. hosts accepts one or more Elasticsearch node/server. Multiple hosts can be defined for failover purposes. When multiple hosts are configured, the events are distributed to these nodes in round robin order. If Elasticsearch is secured, then the credentials can be passed using the username and password settings: output.elasticsearch: enabled: true hosts: ["localhost:9200"] username: "elasticuser" password: "password" To ship the events to the Elasticsearch ingest node pipeline so that they can be pre-processed before being stored in Elasticsearch, the pipeline information can be provided using the pipleline setting: output.elasticsearch: enabled: true hosts: ["localhost:9200"] pipeline: "ngnix_log_pipeline" The default index the data gets written to is of the format metricbeat-%{[beat.version]}-%{+yyyy.MM.dd}. This will create a new index every day. For example if today is December 2, 2017 then all the events are placed in the metricbeat-6.0.0-2017-12-02 index. One can override the index name or the pattern using the index setting. In the following configuration snippet, a new index is created for every month: output.elasticsearch: hosts: ["http://localhost:9200"] index: "metricbeat-%{[beat.version]}-%{+yyyy.MM}" Using the indices setting, one can conditionally place the events in the appropriate index that matches the specified condition. In the following code snippet, if the message contains the DEBUG string, it will be placed in the debug-%{+yyyy.MM.dd} index. If the message contains the ERR string, it will be placed in the error-%{+yyyy.MM.dd} index. If the message contains neither of these texts, then those events will be pushed to the logs-%{+yyyy.MM.dd} index as specified in the index parameter: output.elasticsearch: hosts: ["http://localhost:9200"] index: "logs-%{+yyyy.MM.dd}" indices: - index: "debug-%{+yyyy.MM.dd}" when.contains: message: "DEBUG" - index: "error-%{+yyyy.MM.dd}" when.contains: message: "ERR" When the index parameter is overridden, disable templates and dashboards by adding the following setting in: setup.dashboards.enabled: false setup.template.enabled: false Alternatively, provide the value for setup.template.name and setup.template.pattern in the metricbeat.yml configuration file, or else Metricbeat will fail to run. logstash: It is used to send the events to Logstash. To use Logstash as the output, Logstash needs to be configured with the Beats input plugin to receive incoming Beats events. A sample Logstash output configuration is as follows: output.logstash: enabled: true hosts: ["localhost:5044"] Using the enabled setting, one can enable or disable the output. hosts accepts one or more Logstash servers. Multiple hosts can be defined for failover purposes. If the configured host is unresponsive, then the event will be sent to one of the other configured hosts. When multiple hosts are configured, the events are distributed in random order. To enable load balancing of events across the Logstash hosts, use the loadbalance flag, set to true: output.logstash: hosts: ["localhost:5045", "localhost:5046"] loadbalance: true console: It is used to send the events to stdout. The events are written in JSON format. It is useful during debugging or testing. A sample console configuration is as follows: output.console: enabled: true pretty: true Logging This section contains the options for configuring the Filebeat logging output. The logging system can write logs to syslog or rotate log files. If logging is not explicitly configured, file output is used on Windows systems, and syslog output is used on Linux and OS X. A sample configuration is as follows: logging.level: debug logging.to_files: true logging.files: path: C:logsmetricbeat name: metricbeat.log keepfiles: 10 Some of the configuration options are: level: To specify the logging level. to_files: To write all logging output to files. The files are subject to file rotation. This is the default value. to_syslog: To write the logging output to syslogs if this setting is set to true. files.path, files.name, and files.keepfiles: These are used to specify the location of the file, the name We successfully configured Beat Library, MetricBeat and developed good transmission of operational metrics to Elasticsearch, making it easy to monitor systems and services on servers with much ease. If you found this tutorial useful, do check out the book Learning Elastic Stack 6.0 to examine the fundamentals of Elastic Stack in detail and start developing solutions for problems like logging, site search, app search, metrics and more.      

This tutorial will focus on some of the important architectures present today in deep learning. A lot of the success of neural networks lies in the careful design of the neural network architecture. We will look at the architecture of Autoencoder Neural Networks, Variational Autoencoders, CNN's and RNN's. This tutorial is an excerpt from a book written by Dipanjan Sarkar, Raghav Bali, Et al titled Hands-On Transfer Learning with Python. This book extensively focuses on deep learning (DL) and transfer learning, comparing and contrasting the two with easy-to-follow concepts and examples. Autoencoder neural networks Autoencoders are typically used for reducing the dimensionality of data in neural networks. They are also successfully used for anomaly detection and novelty detection problems. Autoencoder neural networks come under the unsupervised learning category. The network is trained by minimizing the difference between input and output. A typical autoencoder architecture is a slight variant of the DNN architecture, where the number of units per hidden layer is progressively reduced until a certain point before being progressively increased, with the final layer dimension being equal to the input dimension. The key idea behind this is to introduce bottlenecks in the network and force it to learn a meaningful compact representation. The middle layer of hidden units (the bottleneck) is basically the dimension-reduced encoding of the input. The first half of the hidden layers is called the encoder, and the second half is called the decoder. The following depicts a simple autoencoder architecture. The layer named z is the representation layer here: Source: cloud4scieng.org Variational autoencoders The variational autoencoders (VAEs) are generative models and compared to other deep generative models, VAEs are computationally tractable and stable and can be estimated by the efficient backpropagation algorithm. They are inspired by the idea of variational inference in Bayesian analysis. The idea of variational inference is as follows: given input distribution x, the posterior probability distribution over output y is too complicated to work with. So, let's approximate that complicated posterior, p(y | x), with a simpler distribution, q(y). Here, q is chosen from a family of distributions, Q, that best approximates the posterior. For example, this technique is used in training latent Dirichlet allocation (LDAs) (they do topic modeling for text and are Bayesian generative models). Given a dataset, X, VAE can generate new samples similar but not necessarily equal to those in X. Dataset X has N Independent and Identically Distributed (IID) samples of some continuous or discrete random variable, x. Let's assume that the data is generated by some random process, involving an unobserved continuous random variable, z. In this example of a simple autoencoder, the variable z is deterministic and is a stochastic variable. Data generation is a two-step process: A value of z is generated from a prior distribution, ρθ(z) A value of x is generated from the conditional distribution, ρθ(x|z)  So, p(x) is basically the marginal probability, calculated as:   The parameter of the distribution, θ, and the latent variable, z, are both unknown. Here, x can be generated by taking samples from the marginal p(x). Backpropagation cannot handle stochastic variable z or stochastic layer z within the network. Assuming the prior distribution, p(z), is Gaussian, we can leverage the location-scale property of Gaussian distribution, and rewrite the stochastic layer as z = μ + σε , where μ is the location parameter, σ is the scale, and ε is the white noise. Now we can obtain multiple samples of the noise, ε, and feed them as the deterministic input to the neural network. Then, the model becomes an end-to-end deterministic deep neural network, as shown here: Here, the decoder part is same as in the case of the simple autoencoder that we looked at earlier. Types of CNN architectures CNNs are multilayered neural networks designed specifically for identifying shape patterns with a high degree of invariance to translation, scaling, and rotation in two-dimensional image data. These networks need to be trained in a supervised way. Typically, a labeled set of object classes, such as MNIST or ImageNet, is provided as a training set. The crux of any CNN model is the convolution layer and the subsampling/pooling layer. LeNet architecture This is a pioneering seven-level convolutional network, designed by LeCun and their co-authors in 1998, that was used for digit classification. Later, it was applied by several banks to recognize handwritten numbers on cheques. The lower layers of the network are composed of alternating convolution and max pooling layers. The upper layers are fully connected, dense MLPs (formed of hidden layers and logistic regression). The input to the first fully connected layer is the set of all the feature maps of the previous layer: AlexNet In 2012, AlexNet significantly outperformed all the prior competitors and won the ILSVRC by reducing the top-5 error to 15.3%, compared to the runner-up with 26%. This work popularized the application of CNNs in computer vision. AlexNet has a very similar architecture to that of LeNet, but has more filters per layer and is deeper. Also, AlexNet introduces the use of stacked convolution, instead of always using alternative convolution pooling. A stack of small convolutions is better than one large receptive field of convolution layers, as this introduces more non-linearities and fewer parameters. ZFNet The ILSVRC 2013 winner was a CNN from Matthew Zeiler and Rob Fergus. It became known as ZFNet. It improved on AlexNet by tweaking the architecture hyperparameters, in particular by expanding the size of the middle convolutional layers and making the stride and filter size on the first layer smaller, going from 11 x 11 stride 4 in AlexNet to 7 x 7 stride 2 in ZFNet. The intuition behind this was that a smaller filter size in the first convolution layer helps to retain a lot of the original pixel information. Also, AlexNet was trained on 15 million images, while ZFNet was trained on only 1.3 million images: GoogLeNet (inception network) The ILSVRC 2014 winner was a convolutional network called GoogLeNet from Google. It achieved a top-5 error rate of 6.67%! This was very close to human-level performance. The runner up was the network from Karen Simonyan and Andrew Zisserman known as VGGNet. GoogLeNet introduced a new architectural component using a CNN called the inception layer. The intuition behind the inception layer is to use larger convolutions, but also keep a fine resolution for smaller information on the images. The following diagram describes the full GoogLeNet architecture: Visual Geometry Group Researchers from the Oxford Visual Geometry Group, or the VGG for short, developed the VGG network, which is characterized by its simplicity, using only 3 x 3 convolutional layers stacked on top of each other in increasing depth. Reducing volume size is handled by max pooling. At the end, two fully connected layers, each with 4,096 nodes, are then followed by a softmax layer. The only preprocessing done to the input is the subtraction of the mean RGB value, computed on the training set, from each pixel. Pooling is carried out by max pooling layers, which follow some of the convolution layers. Not all the convolution layers are followed by max pooling. Max pooling is performed over a 2 x 2 pixel window, with a stride of 2. ReLU activation is used in each of the hidden layers. The number of filters increases with depth in most VGG variants. The 16-layered architecture VGG-16 is shown in the following diagram. The 19-layered architecture with uniform 3 x 3 convolutions (VGG-19) is shown along with ResNet in the following section. The success of VGG models confirms the importance of depth in image representations: VGG-16: Input RGB image of size 224 x 224 x 3, the number of filters in each layer is circled Residual Neural Networks The main idea in this architecture is as follows. Instead of hoping that a set of stacked layers would directly fit a desired underlying mapping, H(x), they tried to fit a residual mapping. More formally, they let the stacked set of layers learn the residual R(x) = H(x) - x, with the true mapping later being obtained by a skip connection. The input is then added to the learned residual, R(x) + x. Also, batch normalization is applied right after each convolution and before activation: Here is the full ResNet architecture compared to VGG-19. The dotted skip connections show an increase in dimensions; hence, for the addition to be valid, no padding is done. Also, increases in dimensions are indicated by changes in color: Types of RNN architectures An recurrent neural Network (RNN) is specialized for processing a sequence of values, as in x(1), . . . , x(t).  We need to do sequence modeling if, say, we wanted to predict the next term in the sequence given the recent history of the sequence, or maybe translate a sequence of words in one language to another language. RNNs are distinguished from feedforward networks by the presence of a feedback loop in their architecture. It is often said that RNNs have memory. The sequential information is preserved in the RNNs hidden state. So, the hidden layer in the RNN is the memory of the network. In theory, RNNs can make use of information in arbitrarily long sequences, but in practice they are limited to looking back only a few steps. LSTMs RNNs start losing historical context over time in the sequence, and hence are hard to train for practical purposes. This is where  LSTMs (Long short-term memory)  come into the picture! Introduced by Hochreiter and Schmidhuber in 1997, LSTMs can remember information from really long sequence-based data and prevent issues such as the vanishing gradient problem. LSTMs usually consist of three or four gates, including input, output, and forget gates. The following diagram shows a high-level representation of a single LSTM cell: The input gate can usually allow or deny incoming signals or inputs to alter the memory cell state. The output gate usually propagates the value to other neurons as necessary. The forget gate controls the memory cell's self-recurrent connection to remember or forget previous states as necessary. Multiple LSTM cells are usually stacked in any deep learning network to solve real-world problems, such as sequence prediction. Stacked LSTMs If we want to learn about the hierarchical representation of sequential data, a stack of LSTM layers can be used. Each LSTM layer outputs a sequence of vectors rather than a single vector for each item of the sequence, which will be used as an input to a subsequent LSTM layer. This hierarchy of hidden layers enables a more complex representation of our sequential data. Stacked LSTM models can be used for modeling complex multivariate time series data. Encoder-decoder – Neural Machine Translation Machine translation is a sub-field of computational linguistics, and is about performing translation of text or speech from one language to another. Traditional machine translation systems typically rely on sophisticated feature engineering based on the statistical properties of text. Recently, deep learning has been used to solve this problem, with an approach known as Neural Machine Translation (NMT). An NMT system typically consists of two modules: an encoder and a decoder. It first reads the source sentence using the encoder to build a thought vector: a sequence of numbers that represents the sentence's meaning. A decoder processes the sentence vector to emit a translation to other target languages. This is called an encoder-decoder architecture. The encoders and decoders are typically forms of RNN. The following diagram shows an encoder-decoder architecture using stacked LSTMs. Source: tensorflow.org The source code for NMT in TensorFlow is available at Github. Gated Recurrent Units Gated Recurrent Units (GRUs) are related to LSTMs, as both utilize different ways of gating information to prevent the vanishing gradient problem and store long-term memory. A GRU has two gates: a reset gate, r, and an update gate, z, as shown in the following diagram. The reset gate determines how to combine the new input with the previous hidden state, ht-1, and the update gate defines how much of the previous state information to keep. If we set the reset to all ones and update gate to all zeros, we arrive at a simple RNN model: GRUs are computationally more efficient because of a simpler structure and fewer parameters. Summary This article covered various advances in neural network architectures including Autoencoder Neural Networks, Variational Autoencoders,  CNN's and  RNN's architectures. To understand how to simplify deep learning  by taking supervised, unsupervised, and reinforcement learning to the next level using the Python ecosystem, check out this book  Hands-On Transfer Learning with Python Neural Style Transfer: Creating artificial art with deep learning and transfer learning Dr. Brandon explains ‘Transfer Learning’ to Jon 5 cool ways Transfer Learning is being used today

A typical HTTP request To get a better picture of the possible delay incurred when using a web application firewall, it helps to understand the anatomy of a typical HTTP request, and what processing time a typical web page download will incur. This will help us compare any added ModSecurity processing time to the overall time for the entire request. When a user visits a web page, his browser first connects to the server and downloads the main resource requested by the user (for example, an .html file). It then parses the downloaded file to discover any additional files, such as images or scripts, that it must download to be able to render the page. Therefore, from the point of view of a web browser, the following sequence of events happens for each file: Connect to web server. Request required file. Wait for server to start serving file. Download file. Each of these steps adds latency, or delay, to the request. A typical download time for a web page is on the order of hundreds of milliseconds per file for a home cable/DSL user. This can be slower or faster, depending on the speed of the connection and the geographical distance between the client and server. If ModSecurity adds any delay to the page request, it will be to the server processing time, or in other words the time from when the client has connected to the server to when the last byte of content has been sent out to the client. Another aspect that needs to be kept in mind is that ModSecurity will increase the memory usage of Apache. In what is probably the most common Apache configuration, known as "prefork", Apache starts one new child process for each active connection to the server. This means that the number of Apache instances increases and decreases depending on the number of client connections to the server.As the total memory usage of Apache depends on the number of child processes running and the memory usage of each child process, we should look at the way ModSecurity affects the memory usage of Apache. A real-world performance test In this section we will run a performance test on a real web server running Apache 2.2.8 on a Fedora Linux server (kernel 2.6.25). The server has an Intel Xeon 2.33 GHz dual-core processor and 2 GB of RAM. We will start out benchmarking the server when it is running just Apache without having ModSecurity enabled. We will then run our tests with ModSecurity enabled but without any rules loaded. Finally, we will test ModSecurity with a ruleset loaded so that we can draw conclusions about how the performance is affected. The rules we will be using come supplied with ModSecurity and are called the "core ruleset". The core ruleset The ModSecurity core ruleset contains over 120 rules and is shipped with the default ModSecurity source distribution (it's contained in the rules sub-directory). This ruleset is designed to provide "out of the box" protection against some of the most common web attacks used today. Here are some of the things that the core ruleset protects against: Suspicious HTTP requests (for example, missing User-Agent or Accept headers) SQL injection Cross-Site Scripting (XSS) Remote code injection File disclosure We will examine these methods of attack, but for now, let's use the core ruleset and examine how enabling it impacts the performance of your web service. Installing the core ruleset To install the core ruleset, create a new sub-directory named modsec under your Apache conf directory (the location will vary depending on your distribution). Then copy all the .conf files from the rules sub-directory of the source distribution to the new modsec directory: mkdir /etc/httpd/conf/modseccp/home/download/modsecurity-apache/rules/modsecurity_crs_*.conf /etc/httpd/conf/modsec Finally, enter the following line in your httpd.conf file and restart Apache to make it read the new rule files: # Enable ModSecurity core rulesetInclude conf/modsecurity/*.conf Putting the core rules in a separate directory makes it easy to disable them—all you have to do is comment out the above Include line in httpd.conf, restart Apache, and the rules will be disabled. Making sure it works The core ruleset contains a file named modsecurity_crs_10_config.conf. This file contains some of the basic configuration directives needed to turn on the rule engine and configure request and response body access. Since we have already configured these directives, we do not want this file to conflict with our existing configuration, and so we need to disable this. To do this, we simply need to rename the file so that it has a different extension as Apache only loads *.conf files with the Include directive we used above: $ mv modsecurity_crs_10_config.conf modsecurity_crs_10_config.conf.disabled Once we have restarted Apache, we can test that the core ruleset is loaded by attempting to access an URL that it should block. For example, try surfing to http://yourserver/ftp.exe and you should get the error message Method Not Implemented, ensuring that the core rules are loaded. Performance testing basics So what effect does loading the core ruleset have on web application response time and how do we measure this? We could measure the response time for a single request with and without the core ruleset loaded, but this wouldn't have any statistical significance—it could happen that just as one of the requests was being processed, the server started to execute a processor-intensive scheduled task, causing a delayed response time. The best way to compare the response times is to issue a large number of requests and look at the average time it takes for the server to respond. An excellent tool—and the one we are going to use to benchmark the server in the following tests—is called httperf. Written by David Mosberger of Hewlett Packard Research Labs, httperf allows you to simulate high workloads against a web server and obtain statistical data on the performance of the server. You can obtain the program at http://www.hpl.hp.com/research/linux/httperf/ where you'll also find a useful manual page in the PDF file format and a link to the research paper published together with the first version of the tool. Using httperf We'll run httperf with the options --hog (use as many TCP ports as needed), --uri/index.html (request the static web page index.html) and we'll use --num-conn 1000 (initiate a total of 1000 connections). We will be varying the number of requests per second (specified using --rate) to see how the server responds under different workloads. This is what the typical output from httperf looks like when run with the above options: $ ./httperf --hog --server=bytelayer.com --uri /index.html --num-conn1000 --rate 50Total: connections 1000 requests 1000 replies 1000 test-duration20.386 sConnection rate: 49.1 conn/s (20.4 ms/conn, <=30 concurrentconnections)Connection time [ms]: min 404.1 avg 408.2 max 591.3 median 404.5stddev 16.9Connection time [ms]: connect 102.3Connection length [replies/conn]: 1.000Request rate: 49.1 req/s (20.4 ms/req)Request size [B]: 95.0Reply rate [replies/s]: min 46.0 avg 49.0 max 50.0 stddev 2.0 (4samples)Reply time [ms]: response 103.1 transfer 202.9Reply size [B]: header 244.0 content 19531.0 footer 0.0 (total19775.0)Reply status: 1xx=0 2xx=1000 3xx=0 4xx=0 5xx=0CPU time [s]: user 2.37 system 17.14 (user 11.6% system 84.1% total95.7%)Net I/O: 951.9 KB/s (7.8*10^6 bps)Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0 The output shows us the number of TCP connections httperf initiated per second ("Connection rate"), the rate at which it requested files from the server ("Request rate"), and the actual reply rate that the server was able to provide ("Reply rate"). We also get statistics on the reply time—the "reply time – response" is the time taken from when the first byte of the request was sent to the server to when the first byte of the reply was received—in this case around 103 milliseconds. The transfer time is the time to receive the entire response from the server. The page we will be requesting in this case, index.html, is 20 KB in size which is a pretty average size for an HTML document. httperf requests the page one time per connection and doesn't follow any links in the page to download additional embedded content or script files, so the number of such links in the page is of no relevance to our test. Getting a baseline: Testing without ModSecurity When running benchmarking tests like this one, it's always important to get a baseline result so that you know the performance of your server when the component you're measuring is not involved. In our case, we will run the tests against the server when ModSecurity is disabled. This will allow us to tell which impact, if any, running with ModSecurity enabled has on the server. Response time The following chart shows the response time, in milliseconds, of the server when it is running without ModSecurity. The number of requests per second is on the horizontal axis: As we can see, the server consistently delivers response times of around 300 milliseconds until we reach about 75 requests per second. Above this, the response time starts increasing, and at around 500 requests per second the response time is almost a second per request. This data is what we will use for comparison purposes when looking at the response time of the server after we enable ModSecurity. Memory usage Finding the memory usage on a Linux system can be quite tricky. Simply running the Linux top utility and looking at the amount of free memory doesn't quite cut it, and the reason is that Linux tries to use almost all free memory as a disk cache. So even on a system with several gigabytes of memory and no memory-hungry processes, you might see a free memory count of only 50 MB or so. Another problem is that Apache uses many child processes, and to accurately measure the memory usage of Apache we need to sum the memory usage of each child process. What we need is a way to measure the memory usage of all the Apache child processes so that we can see how much memory the web server truly uses. To solve this, here is a small shell script that I have written that runs the ps command to find all the Apache processes. It then passes the PID of each Apache process to pmap to find the memory usage, and finally uses awk to extract the memory usage (in KB) for summation. The result is that the memory usage of Apache is printed to the terminal. The actual shell command is only one long line, but I've put it into a file called apache_mem.sh to make it easier to use: #!/bin/sh# apache_mem.sh# Calculate the Apache memory usageps -ef | grep httpd | grep ^apache | awk '{ print $2 }' | xargs pmap -x | grep 'total kB' | awk '{ print $3 }' | awk '{ sum += $1 } END { print sum }' Now, let's use this script to look at the memory usage of all of the Apache processes while we are running our performance test. The following graph shows the memory usage of Apache as the number of requests per second increases: Apache starts out consuming about 300 MB of memory. Memory usage grows steadily and at about 150 requests per second it starts climbing more rapidly. At 500 requests per second, the memory usage is over 2.4 GB—more than the amount of physical RAM of the server. The fact that this is possible is because of the virtual memory architecture that Linux (and all modern operating systems) use. When there is no more physical RAM available, the kernel starts swapping memory pages out to disk, which allows it to continue operating. However, since reading and writing to a hard drive is much slower than to memory, this starts slowing down the server significantly, as evidenced by the increase in response time seen in the previous graph. CPU usage In both of the tests above, the server's CPU usage was consistently around 1 to 2%, no matter what the request rate was. You might have expected a graph of CPU usage in the previous and subsequent tests, but while I measured the CPU usage in each test, it turned out to run at this low utilization rate for all tests, so a graph would not be very useful. Suffice it to say that in these tests, CPU usage was not a factor. ModSecurity without any loaded rules Now, let's enable ModSecurity—but without loading any rules—and see what happens to the response time and memory usage. Both SecRequestBodyAccess and SecResponseBodyAccess were set to On, so if there is any performance penalty associated with buffering requests and responses, we should see this now that we are running ModSecurity without any rules. The following graph shows the response time of Apache with ModSecurity enabled: We can see that the response time graph looks very similar to the response time graph we got when ModSecurity was disabled. The response time starts increasing at around 75 requests per second, and once we pass 350 requests per second, things really start going downhill. The memory usage graph is also almost identical to the previous one: Apache uses around 1.3 MB extra per child process when ModSecurity is loaded, which equals a total increase of memory usage of 26 MB for this particular setup. Compared to the total amount of memory Apache uses when the server is idle (around 300 MB) this equals an increase of about 10%. Mod Security with the core ruleset loaded Now for the really interesting test we'll run httperf against ModSecurity with the core ruleset loaded and look at what that does to the response time and memory usage. Response time The following graph shows the server response time with the core ruleset loaded: At first, the response time is around 340 ms, which is about 35 ms slower than in previous tests. Once the request rate gets above 50, the server response time starts deteriorating. As the request rates grows, the response time gets worse and worse, reaching a full 5 seconds at 100 requests per second. I have capped the graph at 100 requests per second, as the server performance has already deteriorated enough at this point to allow us to see the trend. We see that the point at which memory usage starts increasing has gone down from 75 to 50 requests per second now that we have enabled the core ruleset. This equals a reduction in the maximum number of requests per second the server can handle of 33%.