How-To Tutorials

This article is an excerpt from the book, Machine Learning with Microsoft Power BI, by Greg Beaumont. This book is designed for data scientists and BI professionals seeking to improve their existing solutions and workloads using AI.Data description generation plays a vital role in understanding complex datasets, but it can be a time-consuming task. Enter ChatGPT, an advanced AI language model developed by OpenAI. Trained on extensive text data, ChatGPT demonstrates impressive capabilities in understanding and generating human-like responses. In this article, we explore how ChatGPT can revolutionize data analysis by expediting the creation of accurate and coherent data descriptions. We delve into its training process, architecture, and potential applications in fields like research, journalism, and business analytics. While acknowledging limitations, we unveil the transformative potential of ChatGPT for data interpretation and knowledge dissemination. Our first step will be to identify a suitable use case for leveraging the power of GPT models to generate descriptions of elements of FAA Wildlife Strike data. Our objective is to unlock the potential of external data by creating prompts for GPT models that can provide detailed information and insights about the data we are working with. Through this use case, we will explore the value that GPT models can bring to the table when it comes to data analysis and interpretation.For example, a description of the FAA Wildlife Strike Database by ChatGPT might look like this: Figure 1 – OpenAI ChatGPT description of FAA Wildlife Strike Database Within your solution using the FAA Wildlife Strike database, you have data that could be tied to external data using the GPT models. A few examples include additional information about:AirportsFAA RegionsFlight OperatorsAircraftAircraft EnginesAnimal SpeciesTime of Year When the scoring process for a large number of separate rows in a dataset is automated, we can use a GPT model to generate descriptive text for each individual row. It is worth noting that ChatGPT's approach varies from this, as it operates as a chatbot that calls upon different GPT models and integrates past conversations into future answers. Despite the differences in how GPT models will be used in the solution, ChatGPT can still serve as a valuable tool for testing various use cases.When using GPT models, the natural language prompts that are used to ask questions and give instructions will impact the context of the generated text. Prompt engineering is a topic that has surged in popularity for OpenAI and LLMs. The following prompts will provide different answers when using “dogs” as a topic for a GPT query:Tell me about dogs:From the perspective of an evolutionary biologist, tell me about dogs:Tell me the history of dogs:At a third-grade level, tell me about dogs:When planning for your use of OpenAI on large volumes of data, you should test and evaluate your prompt engineering strategy. For this book, the use cases will be kept simple since the goal is to teach tool integration with Power BI. Prompt engineering expertise will probably be the topic of many books and blogs this year. You can test different requests for a description of an FAA Region in the data: Figure 2 – Testing the utility of describing an FAA Region using OpenAI ChatGPT You can also combine different data elements for a more detailed description. The following example combines data fields with a question to ask “Tell me about the Species in State in Month”: Figure 3 – Using ChatGPT to test a combination of data about Species, State, and Month There are many different options to consider. To combine a few fields of data and provide useful context about the data, you decide to plan a use case for describing the aircraft and operator. An example can be tested with the following formula in OpenAI ChatGPT such as “Tell me about the airplane model Aircraft operated by the Operator in three sentences." Here is an example using data from a single row of the FAA Wildlife Strike database: Figure 4 – Information about an airplane in the fleet of an operator as described by OpenAI ChatGPT From a prompt engineering perspective, asking this question for multiple reports in the FAA Wildlife Strike database would require running the following natural language query on each row of data (column names are depicted with brackets): Tell me about the airplane model [Aircraft] operated by [Operator] in three sentences:SummaryThis article explores how ChatGPT expedites the generation of accurate and coherent data descriptions. Unveiling its training process, architecture, and applications in research, journalism, and business analytics, we showcase how ChatGPT revolutionizes data interpretation and knowledge dissemination. Acknowledging limitations, we highlight the transformative power of this AI technology in enhancing data analysis and decision-making. Author BioGreg Beaumont is a Data Architect at Microsoft; Greg is an expert in solving complex problems and creating value for customers. With a focus on the healthcare industry, Greg works closely with customers to plan enterprise analytics strategies, evaluate new tools and products, conduct training sessions and hackathons, and architect solutions that improve the quality of care and reduce costs. With years of experience in data architecture and a passion for innovation, Greg has a unique ability to identify and solve complex challenges. He is a trusted advisor to his customers and is always seeking new ways to drive progress and help organizations thrive. For more than 15 years, Greg has worked with healthcare customers who strive to improve patient outcomes and find opportunities for efficiencies. He is a veteran of the Microsoft data speaker network and has worked with hundreds of customers on their data management and analytics strategies.You can follow Greg on his LinkedIn 

What if you created something that allowed people to make fun of you? Such is the life and career of Jack Dorsey, co-founder and CEO of Twitter. Yesterday the tech guru did a TED Talk public conversation with the head of TED, Chris Anderson, and Whitney Pennington Rodgers, TED’s current affairs curator. The major part of the whole conversation involved around the recent “concerns and opportunities for Twitter’s future.” For most of the interview, Dorsey outlined steps that Twitter has taken to combat abuse and misinformation, and Anderson explained why the company’s critics sometimes find those steps insufficient and unsatisfying. He compared Twitter to the Titanic, and Dorsey to the captain, listening to passengers’ concerns about the iceberg up ahead — then going back to the bridge and showing “this extraordinary calm.” “It’s democracy at stake, it’s our culture at stake,” Anderson said, echoing points made in a talk by British journalist Carole Cadwalladr. So why isn’t Twitter addressing these issues with more urgency? “We are working as quickly as we can, but quickness will not get the job done,” Dorsey replied to Anderson. “It’s focus, it’s prioritization, it’s understanding the fundamentals of the network.” Jack further argued that while Twitter could “do a bunch of superficial things to address the things you’re talking about,” that isn’t the real solution. “We want the changes to last, and that means going really, really deep,” Dorsey said. In his view, that means rethinking how Twitter incentivizes user behavior. He suggested that the service works best as an “interest-based network,” where you log in and see content relevant to your interests, no matter who posted it — rather than a network where everyone feels like they need to follow a bunch of other accounts, and then grow their follower numbers in turn. Dorsey recalled that when the team was first building the service, it decided to make follower count “big and bold,” which naturally made people focus on it. “Was that the right decision at the time? Probably not,” he said. “If I had to start the service again, I would not emphasize the follower count as much... Since he isn’t starting from scratch, Dorsey suggested that he’s trying to find ways to redesign Twitter to shift the “bias” away from accounts and toward interests. He also pointed to efforts that Twitter has already announced to measure (and then improve) conversational health and to use machine learning to automatically detect abusive content. In terms of how the company is currently measuring its success, Dorsey said it focuses primarily on daily active users, and secondly on “conversation chains — we want to incentivize healthy contributions back to the network.” In contradiction to his response when over the weekend, Ilhan Omar an immigrant, and a Black Muslim representing the state of Minnesota in the US House of representatives —reported an increase in death threats. The threats started appearing after President Trump tweeted out a video that intercut a speech she had given with footage of the 9/11 attacks. Many of the threats were made on Twitter. Then on Monday, as Notre Dame burned, people came to the platform to mourn the loss in real time, but also to spread lies and hate as quickly as the flames engulfed the cathedral’s spire. When Omar tweeted her own heartfelt condolences, people replied with more death threats. Dorsey didn’t address any of these incidents specifically at TED. In fact, his answers lacked specificity overall. One Twitter user, Chad Loder created a Twitter moment to stand with Ilhan Omar. He asked users to report all the tweets with death threats to Ilhan. To this Twitter safety team removed all the tweets in just 30 minutes and he could not further add any tweets. https://twitter.com/chadloder/status/1118287939220336640 When Jack was asked pointed questions, he evaded them, as he often does. Rodgers further asked him how many people are working on content moderation on Twitter—a number the company has never published. “It varies,” Dorsey replied. “We want to be flexible on this. There are no amount of people that can actually scale this, which is why we have done so much work on proactively taking down abuse.” Dorsey further announced that a year ago, Twitter wasn’t proactively monitoring abuse actively using machine learning at all. Instead, it relied entirely on human reporting—a burden Dorsey later had to recognized was unfairly put on the victims of the abuse. “We’ve made progress,” he said. “Thirty-eight percent of abusive tweets are now proactively recognized by machine-learning algorithms, but those that are recognized are still reviewed by humans. But that was from zero percent just a year ago.” As he uttered those words, Twitter sent out a press release with more information on the effort, highlighting that three times more abusive accounts are being suspended within 24 hours of getting reported compared with this time last year. The progress of 38% is not exactly a lot when Facebook’s most recent transparency report, says that over 51% of content on hate speech was flagged before users reported it. Nor did Dorsey or the official Twitter announcement provide many details about how the technology to proactively flag abuse works. While the conversations continued, they came up with an idea to invite Twitter users to tweet questions via the hashtag #AskJackAtTED, which was then projected on a big screen behind Dorsey. Looking at all the tweets coming up live it seemed to be a really bad idea. https://twitter.com/TEDTalks/status/1118182843354701826 In Twitter spirit, sarcastic piercing tweets soon overflowed on the screen asking Dorsey crisp pointed questions about long pending failures to make the platform a safe place for healthy conversation. Users charged Dorsey left, right and centre with questions. Carole Cadwalladr asked why a video that showed her being beaten up & threatened with a gun to soundtrack of Russian anthem stayed up for 72 hours despite 1000s of complaints. Other came up saying how do you prioritize healthy conversations without silencing protest on your platform? https://twitter.com/carolecadwalla/status/1118184265781587968 https://twitter.com/jonnysun/status/1118190191066398720 Tweets continued pouring in with questions like why is women harassment on the platform ignored for decades and why do you still allow Nazi groups on Twitter. https://twitter.com/beccalew/status/1118194017840144386 https://twitter.com/chick_in_kiev/status/1118208375534362624 One of them asked if there was anyway to add a fact-check function to viral tweets which involves misinformation or conspiracy theories. https://twitter.com/MollyJongFast/status/1118185171860303873 Another popular tweet said it wants Twitter to create a user bill where they have the right to be free of harassment and the right to safety. https://twitter.com/RedTRaccoon/status/1118256403498577920 https://twitter.com/olmaverick/status/1118186175364075521 https://twitter.com/baratunde/status/1118186537215094785 https://twitter.com/johnbattelle/status/1118185819691528197 https://twitter.com/MaraWilson/status/1118190714494410752 While some users said that Twitter is of no value at all others raised questions on why does the platform allow the organizers of Charlottesville disaster amplify their message and find new recruits. Another added why is President Donald Trump allowed to violate Twitter policies by spreading hateful and inflammatory tweets against Ilhan Omar. https://twitter.com/Bro_Pair/status/1118186479048478725 https://twitter.com/WajahatAli/status/1118183456985448450 https://twitter.com/brosandprose/status/1118185344397082625 Eventually the tweets disappeared from the screen and someone from the audience made a suspicion that it was always a plan to ditch in between. On the other hand Anderson summed it up sarcastically: “We are on this great voyage with you on a ship, and there are people on board in steerage who are expressing discomfort and you, unlike other captains are saying, ‘well, tell me, I want to hear,’ and they’re saying ‘we’re worried about the iceberg ahead,'” Anderson says. “And you say, ‘our ship hasn’t been built for steering as well as it might,’ and you’re showing this extraordinary calm!” Highlights from Jack Dorsey’s live interview by Kara Swisher on Twitter: on lack of diversity, tech responsibility, physical safety and more Russia opens civil cases against Facebook and Twitter over local data laws Twitter memes are being used to hide malware

I believe the most significant leap in understanding networking comes from learning how to turn a simple home network into a full-featured small business environment.  Surprisingly, I’ve never come across a plain-language tutorial on how to do just that.  However, I’ve learned a great deal throughout my career so far and today, I intend to write the article I wish I had found several years ago. Prerequisites Cable/DSL/FIOS Modem—Any internet connection device that has Ethernet out will do. Wireless Router or Switch—Either will do. Be sure to read the next section for clarification. Firewall Computer —The key to a business type environment is a dedicated hardware firewall.  We’ll be using pfSense, and if you check the hardware requirements you’ll see that an older machine will do nicely.  Any machine to be configured as a firewall requires a minimum of two Ethernet ports. Client Computer—A laptop or desktop computer to be used as a client.  Windows or Linux will both work, but I’ll only be referencing Windows commands in this tutorial. Disable All Security Software on All Client Machines—Make sure you disable Windows Firewall and any other security software you may have.  This introduces an added level of software security that will only hang you up while you’re trying to establish that your hardware firewall is working correctly. Network Device Clarification It’s very important to understand the similarities and differences between the core types of networking equipment.  The following should shed some light on the subject, in order of simplest to most complex: Network Interface Card—Every type of wired networking device has at least 1 NIC, with 1 Ethernet port. A NIC can have multiple Ethernet ports. Each Ethernet port has a unique MAC address. Cross-Over Cable—A crossover cable is sometimes needed to connect two network devices together without a switch. This may be the case if you’re using older hardware, and you want to connect a single client computer to your firewall computer. Most of the newer hardware “autosenses” regular and crossover cables so both will work automatically, but if you’re not receiving a signal or seeing any flashing lights then try using a crossover cable. Switch—A switch adds more ports to your network and eliminates the need for a crossover cable. A typical 8-port switch will have 1 Ethernet cable running to the firewall, and the other 7 cables running to clients or other network devices.  It’s important to remember that it doesn’t matter which cable goes in to which port, they are all the same. Router—A router provides an interface between two (or more) networks and will also usually act as a DHCP server. That's why on a router you have a special WAN port for that huge network we call the internet, and then a port for your internal Local Area Network (LAN). As a DHCP server, it’s responsible for automatically assigning an IP address to any device connected to the LAN and then it routes all traffic between both networks. Wireless Access Point—A wireless access point (WAP) is simply a wireless switch. When you connect to one, it’s the same as if you would have plugged your computer into a switch. "Wireless Router"—I have put this item in quotes because it’s important to understand that this type of device that most of us own is usually a combination of all the devices above. It has built-in NICs with Ethernet ports. It auto-senses crossover connections so they are never an issue. It routes traffic from that special WAN (usually labeled “Internet”) port to other ports for you to use as a LAN. Those 4 other ports are all for the same single subnet (your LAN) so that means it’s actually a 4-port switch. Additionally, those antennas provide WAP functionality. Acronym Reference WAN—Wide Area Network. Also known as the internet. LAN—Local Area Network. Your internal network, also known as your domain or your intranet. DMZ—De-Militarized Zone. A fancy name for just another type of internal network just like your LAN. The difference is, using firewall rules you prevent any traffic that comes into your DMZ subnet from going to your LAN subnet, for security purposes. This is where you would host a web server or FTP server, a place where anyone on the internet can access certain things without having access to your private LAN devices. DHCP—A type of service that automatically hands out IP addresses. Many types of network devices are configurable as DHCP servers. WAP—Wireless Access Point.  Essentially, a wireless switch. NIC—Network Interface Card. * WAN’s, LAN’s  and DMZ’s are all the same type of Ethernet network.  They all use the same hardware and work in the same way.  They are just given special names to differentiate how these networks are used. Part 1 – Understanding your Existing Home Network The diagram above is very typical of home networks. Study the diagram carefully and note the following key points of interest: The wireless router can connect clients wirelessly using its built-in WAP. The wireless router can connect wired clients using its built-in 4-port switch. While not noticeable in the diagram, those wired clients could be connected via crossover cable (as long as the client’s NIC’s also support "autosensing"). And again, while not noticeable, the wireless router is likely configured as a DHCP server and will automatically hand out (and keep track of) IP addresses for each client that connects to it. Unfortunately, this diagram fails to illustrate the importance of subnets. Also referred to as interfaces, subnets are the keystones of understanding how to take your network to the next level: This illustration highlights a few more key benefits and limitations of a typical home network: The wireless router is the central piece of the network. The wireless router provides a ton of functionality for a single network device.  It’s a router, autosensing switch, WAP, DHCP server and most even have very limited firewall functionality (allow/block IP’s, website filtering, port-forwarding, etc). The wireless router is limited to two interfaces, the WAN and the LAN. So how do we resolve these limitations? How do we bump our network up a notch? We replace that wireless router with a serious hardware firewall. We take an old computer we have laying around and we turn it into router on steroids. Enter pfSense. Part 2 – Creating a Small Business Network Now that we thoroughly understand what we’ve already got, let’s take a look at what we want to create: As you can see in the diagram above, our dedicated firewall is the centerpiece of our small business network.  Any firewall requires a minimum of two different interfaces, which means it requires a minimum of two NICs.  It should be obvious that the network illustrated above has four, but we’ll only be using two for this tutorial. Books from Packt Cacti 0.8 Network Monitoring FreePBX 2.5 Powerful Telephony Solutions Building Enterprise Ready Telephony Systems with sipXecs 4.0 Asterisk 1.4 – the Professional’s Guide ASP.NET 3.5 CMS Development Choosing an Open Source CMS: Beginner's Guide 3D Game Development with Microsoft Silverlight 3: Beginner's Guide Magento: Beginner's Guide  

Generative Adversarial Models, introduced by Ian Goodfellow, are the next big revolution in the field of deep learning. Why? Because of their ability to perform semi-supervised learning where there is a vast majority of data is unlabelled. Here, GANs can efficiently carry out image generation tasks and other tasks such as converting sketches to an image, conversion of satellite images to a map, and many other tasks. GANs are capable of generating realistic images in any circumstances, for instance, giving some text written in a particular handwriting as an input to the generative model in order to generate more texts in the similar handwriting. The speciality of these GANs is that as compared to discriminative models, these generative models make use of a joint distribution probability to generate more likely samples. In short, these generative models or GANs are an improvisation to the discriminative models. Let’s explore some of the research papers that are contributing to further advancements in GANs. CycleGAN: Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks This paper talks about CycleGANs, a class of generative Adversarial networks that carry out Image-to-Image translation. This means, capturing special characteristics of one image collection and figuring out how these characteristics could be translated into the other image collection, all in the absence of any paired training examples. CycleGANs method can also be applied in variety of applications such as Collection Style Transfer, Object Transfiguration, season transfer and photo enhancement. Cycle GAN architecture Source: GitHub CycleGANs are built upon the advantages of PIX2PIX architecture. The key advantage of CycleGANs model is, it allows to point the model at two discrete, unpaired collection of images.For example, one image collection say Group A, would consist photos of landscapes in summer whereas Group B would include photos of  landscapes in winter. The CycleGAN model can learn to translate the images between these two aesthetics without the need to merge tightly correlated matches together into a single X/Y training image. Source: Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks The way CycleGANs are able to learn such great translations without having explicit X/Y training images involves introducing the idea of a full translation cycle to determine how good the entire translation system is, thus improving both generators at the same time. Source: Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks Currently, the applications of CycleGANs can be seen in Image-to-Image translation and video translations. For example they can be seen used in Animal Transfiguration, Turning portrait faces into doll faces, and so on. Further ahead, we could potentially see its implementations in audio, text, etc., would help us in generating new data for training. Although this method has compelling results, it also has some limitations The geometric changes within an image are not fully successful (for instance, the cat to dog transformation showed minute success). This could be caused by the generator architecture choices, which are tailored for good performance on the appearance changes. Thus, handling more varied and extreme transformations, especially geometric changes, is an important problem. Failure caused by the distribution characteristics of the training datasets. For instance, in the horse to zebra transfiguration, the model got confused as it was trained on the wild horse and zebra synsets of ImageNet, which does not contain images of a person riding a horse or zebra. These and some other limitations are described in the research paper. To read more about CycleGANs in detail visit the link here. Wasserstein GAN In this paper, we get an exposure to Wasserstein GANs and how they overcomes the drawbacks in original GANs. Although GANs have shown a drastic success in realistic image generation, the training however is not that easy as the process is slow and unstable. In the paper proposed for WGANs, it is empirically shown that WGANs cure the training problem. Wasserstein distance, also known as Earth Mover’s (EM) distance, is a measure of distance between two probability distributions. The basic idea in WGAN is to replace the loss function so that there always exists a non-zero gradient. This can be done using Wasserstein distance between the generator distribution and the data distribution. Training these WGANs does not require keeping a balance in training of the discriminator and the generator. It also doesn’t require a design of the network architecture too. One of the most fascinating practical benefits of WGANs is the ability to continuously estimate the EM distance by training the discriminator to an optimal level. The learning curves when used for plotting are useful for debugging and hyperparameter searches. These curves also correlate well with the observed sample quality and improved stability of the optimization process. Thus, Wasserstein GANs are an alternative to traditional GAN training with features such as: Improvement in the stability of learning Elimination of problems like mode collapse Provide meaningful learning curves useful for debugging and hyperparameter searches Furthermore, the paper also showcases that the corresponding optimization problem is sound, and provides extensive theoretical work highlighting the deep connections to other distances between distributions. The Wasserstein GAN has been utilized to train a language translation machine. The condition here is that there is no parallel data between the word embeddings between the two languages. Wasserstein GANs have been used to perform English-Russian and English-Chinese language mappings. Limitations of WGANs: WGANs suffer from unstable training at times, when one uses a momentum based optimizer or when one uses high learning rates. Includes slow convergence after weight clipping, especially when clipping window is too large. It also suffers from the vanishing gradient problem when the clipping window is too small. To have a detailed understanding of WGANs have a look at the research paper here. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets This paper describes InfoGAN (Information-theoretic extension to the Generative Adversarial Network). It can learn disentangled representations in a completely unsupervised manner. In traditional GANs, learned dataset is entangled i.e. encoded in a complex manner within the data space. However, if the representation is disentangled, it would be easy to implement and easy to apply tasks on it. InfoGAN solves the entangled data problem in GANs.   Specifically, InfoGAN successfully disentangles writing styles from digit shapes on the MNIST dataset, extracts poses of objects correctly irrespective of the lighting conditions within the 3D rendered images, and background digits from the central digit on the SVHN dataset. It also discovers visual concepts that include hairstyles, presence/absence of eyeglasses, and emotions on the CelebA face dataset. InfoGAN does not require any kind of supervision. In comparison to InfoGAN, the only other unsupervised method that learns disentangled representations is hossRBM, a higher-order extension of the spike-and-slab restricted Boltzmann machine which disentangles emotion from identity on the Toronto Face Dataset. However, hossRBM can only disentangle discrete latent factors, and its computation cost grows exponentially in the number of factors. Whereas, InfoGAN can disentangle both discrete and continuous latent factors, scale to complicated datasets, and typically requires no more training time than regular GAN. In the experiments given in the paper, firstly the comparison of  InfoGAN with prior approaches on relatively clean datasets is shown. Another experiment shown is, where InfoGAN can learn interpretable representations on complex datasets (here no previous unsupervised approach is known to learn representations of comparable quality.) Thus, InfoGAN is completely unsupervised and learns interpretable and disentangled representations on challenging datasets. Additionally, InfoGAN adds only negligible computation cost on top of GAN and is easy to train. The core idea of using mutual information to induce representation can be applied to other methods like VAE (Variational AutoEncoder) in future. The other possibilities with InfoGAN in future could be,learning hierarchical latent representations, improving semi-supervised learning with better codes, and using InfoGAN as a high-dimensional data discovery tool. To know more about this research paper in detail, visit the link given here. Progressive growing of GANs for improved Quality, Stability, and Variation This paper describes a brand new method for training your Generative Adversarial Networks. The basic idea here is to train both the generator and the discriminator progressively. This means, starting from a low resolution and adding new layers so that the model increases in providing images with finer details as training progresses. Such a method speeds up the training and also stabilizes it to a greater extent, which in turn produces images of unprecedented quality. For instance, a higher quality version of the CELEBA images dataset that provides output resolutions up to 10242 pixels.   Source: https://arxiv.org/pdf/1710.10196.pdf When new layers are added to the networks, they fade in smoothly. This helps in avoiding the sudden shocks to the already well-trained, smaller resolution layers. Also, the progressive training has various other benefits. The generation of smaller images is substantially more stable because there is less class information and fewer modes By increasing the resolution little by little, we are continuously asking a much simpler question compared to the end goal of discovering a mapping from latent vectors to e.g. 10242 images Progressive growing of GANs also reduces the training time. In addition to this, most of the iterations are done at lower resolutions, and the quality of the result obtained is upto 2-6 times faster, depending on the resolution of the final output. Thus, by progressively training GANs results into better quality, stability, and variation in images. This may also lead to true photorealism in near future. The paper concludes with the fact that, though there are certain limitations with this training method, which include semantic sensibility and understanding dataset-dependent constraints(such as certain objects being straight rather than curved). This leaves a lot to be desired from GANs and there is also room for improvement in the micro-structure of the images. To have a thorough understanding of this research paper, read the paper here.  

For example, let's say we have a network of four Linux servers, two Unix servers, and one Cisco router. Here, if we use a template, we will need to make only three different templates: one for the Linux servers, one for the Unix servers, and one for the Cisco router. You may ask why we have to make a template for the Cisco router? We will make it so that we can use it later. Cacti templates can be imported and exported via the Console under Import/Export. You can only import templates that have been created on a system that is at the same, or an earlier, version of Cacti. At the end of this article, there is a list of third-party templates that can be imported. Types of Cacti templates Cacti templates are broken into two areas: Graph templates Host templates In the above figure, under the Templates section, you will see the three types of templates that come with the Cacti basic installation. If you click on one of those links, you will get the complete list of templates for that type. Graph templates Graphs are used to visualize the data you have collected. A graph template provides a skeleton for an actual graph. When you have more then one system/device, a graph template will save you lots of time and also reduce the possibility of error. Any parameters defined within a graph template are copied to all the graphs that are created using this template. Creating a graph template Now, we are going to create a new graph template. Under the Templates heading, click on Graph Templates. A list of the already available graph templates will be shown. Click on the Add link in the top right corner. You will get a screen like this: Here, you have to give the values that will be used in future to create the graph. If you have checked Use Per-Graph Value (Ignore this Value), then every time while using this graph template to create a graph, you have to give an input for this option. It's always best to enable this option for title field. After filling all these fields, click on the Create button. The graph template will be created. Now, we need to add a Graph Template Item and Graph Item Inputs to complete this graph template. Graph Template Item Graph template items are the various items that will be shown on the graph. To add a graph template item, click on Add on the right side of the Graph Template Items box. You will get this page. The following are the fields that can be filled in: When creating a graph item, you must always start with an AREA item before using STACK; otherwise, your graph will not render. Graph Item Inputs The second box is Graph Item Inputs. Graph item inputs are the input source through which the data will be collected. To add a new graph item input, click on the Add link on the right side of the Graph Item Inputs box. Below are the various fields that have to be filled out for a graph input item: After completing all these fields, click on the Create button. Do this again to add more graph item inputs to this graph template.

Getting iAd Producer iAd Producer is the tool that allows us to assemble great interactive ads with a simple drag-and-drop visual interface. Download and install iAd Producer on your Mac, so that you can start creating an ad. Time for action – installing iAd Producer To install iAd Producer, follow these steps: To download and use iAd Producer, you need to be a paid member of the iOS Developer Program. Go to https://developer.apple.com/ios/ and click on the Log in button. Enter your Apple ID and password, and click on Sign In. After you've signed in, find the Downloads section at the bottom of the page. Click on iAd Producer to start downloading it. You can see the download highlighted here: If you cannot see iAd Producer in the Downloads, make sure you're logged in and your developer account has been activated. After the download is complete, open the file and run iAd Producer.mpkg to start the installation wizard. Follow the steps in the installation and enter your Mac password, if asked for it. When installing certain software, you need to enter your Mac password to allow it to have privileged access to your system. Don't confuse this with your Apple ID that we set up for the iOS Developer Program. If you don't have a password on your Mac, just leave the password area blank and click on OK. When you've gone through the installation steps it'll take a couple of moments to install. After you get a The installation was successful message you can close the installer. What just happened? We now have iAd Producer installed; whenever you need to open it, you can find it in the Applications folder on your Mac. Working with iAd Producer Let's take a look at some of the main parts of iAd Producer that you'll be using regularly, to familiarize yourself with the interface. Launch screen When you first open iAd Producer, you'll be able to start a new iPhone or iPad project from the project selector, as shown in the following screenshot. As the screen size and experience is so different between the two devices, we have to design and build ads specifically for each one: From the launch screen, you can also open existing projects you've been working on. Default ad Once you have chosen to create either an iPad or iPhone iAd, a placeholder ad is created for you, showing the visual flow. This is the overview of your ad, which you'll be using to piece the sections of your ad together. The following screenshot shows the default overview: Double-clicking on any of the screens in your ad flow will ask you to pick a template for that page; once assigned, you're then able to design the iAd using the canvas editor. Template selector Before we edit any page of an ad, we have to apply a template to it, even if it's just a blank canvas to build upon. iAd Producer automatically shows the relevant templates to the current page you're editing. This means your ad follows a structure that the users expect. Templates provide some great starting points for your iAd, whether it's for a simple banner with an image and text or a 3D image carousel that the user can flick and manipulate, all created with easy point and click. The following screenshot is an example of the template chooser: Asset Library The Asset Library holds all the media and content for your iAd, such as the images, videos, and audio. When adding media to your Asset Library, make sure you're using high-resolution images for the high-resolution Retina display. iAd Producer automatically generates the lower-resolution images for your ad, whenever you import resources. If you wanted an image to be 200px wide and 300px high, you should double the horizontal and vertical pixels to 400px wide and 600px high. This will mean your graphics look crisp and awesome on the high-resolution screens. The following screenshot shows an example of media in the Asset Library: Ad canvas Once you've selected a template, you can double-click on the item in the Overview to open up the canvas for that page. The ad canvas is where you customize your iAd with a powerful visual editor to manipulate each page of your ad. Here's an example of the ad canvas with a video carousel added to it: Setting up your ad Let's create and save an empty project to use as we create our iAd; you'll only need to do this once for each ad. Whenever you're working with something digital, it's important to save your iAd whenever you make a significant change, in case iAd Producer closes unexpectedly. Try to get into the habit of saving regularly, to avoid losing your ad. Time for action – creating a new project In order to create a new project, follow the ensuing steps: If you haven't created a new project already, open iAd Producer from your Applications folder. Select the iPhone from the launch screen and choose Select. You'll now see the default ad overview. iAd Producer has automatically made us a project called Untitled and populated it with the default set of pages. From the File menu, select Save to save your empty iAd, ready to have the components added to it later. Name the project something like Dino Stores, as that is the ad we'll be working on. You can now save the progress of your project at any time by choosing File then Save from the menu bar or pressing Command + S on your keyboard. What just happened? You've now seen the project selector and the launch screen in action, and have the base project that we'll be building upon as we make our first iAd. If you quit this project you can now open the project from within iAd Producer by clicking on File | Open, from the menu bar; or, simply double-click the project file in Finder to automatically open it. Getting the resources In this article, we'll be using the Dino Stores example resources that are available to download with this book. If you want to use your own assets, you'll need the following media: An image for your banner, approximately 120px wide and 100px high An image of your company logo or name, around 420px wide and 45px high An 80px square image, with transparency, to be used as a map pin A loading image, approximately 600px wide and 400px high Between six and 10 images for a gallery, each around 304px wide and 440px high Two or more images that will change when the iPhone is shaken, each around 600px wide and 800px high An image related to your product or service, at least 300px wide, to use on the main menu page These pixel sizes are at double-size to account for the high-resolution Retina display found on the iPhone 4 and later. iAd Producer will automatically create the lower-resolution versions for older devices.

What makes a game? We saw that majority of the games created on the CryENGINE SDK have historically been first-person shooters containing a mix of sandbox and directed gameplay. If you have gone so far as to purchase a book on the use of the CryENGINE 3 SDK, then I am certain that you have had some kind of idea for a game, or even improvements to existing games, that you might want to make. It has been my experience professionally that should you have any of these ideas and want to share or sell them, the ideas that are presented in a playable format, even in early prototype form, are far more effective and convincing than any PowerPoint presentation or 100-page design document. Reducing, reusing, recycling Good practice when creating prototypes and smaller scale games, especially if you lack the expertise in creating certain assets and code, is to reduce, reuse, and recycle. To break down what I mean: Reduce the amount of new assets and new code you need to make Reuse existing assets and code in new and unique ways Recycle the sample assets and code provided, and then convert them for your own uses Developing out of the box As mentioned earlier, the CryENGINE 3 SDK has a huge amount of out-of-the-box features for creating games. Let's begin by following a few simple steps to make our first game world. Before proceeding with this example, it's important to understand the features it is displaying; the level we will have created by the end of this article will not be a full, playable game, but rather a unique creation of yours, which will be constructed using the first major features we will need in our game. It will provide an environment in to which we can design gameplay. With the ultimate goal of this article being to create our own level with the core features immediately available to us, we must keep in mind that these examples are orientated to compliment a first-person shooter and not other genres. The first-person shooter genre is quite well defined as new games come out every year within this genre. So, it should be fairly easy for any developer to follow these examples. In my career, I have seen that you can indeed accomplish a good cross section of different games with the CryENGINE 3 SDK. However, the third- and first-person genres are significantly easier to create, immediately with the example content and features available right out of the box. For the designers:This article is truly a must-have for designers working with the engine. Though, I would highly recommend that all users of sandbox know how to use these features, as they are the principal features typically used within most levels of the different types of games in the CryENGINE. Time for action - creating a new level Let's follow a few simple steps to create our own level: Start the Editor.exe application. Select File | New. This will present you with a New Level dialog box that allows you to do the adjustments of some principal properties of your masterpiece to come. The following screenshot shows the properties available in New Level: Name this New Level, as Book_Example_1. The name that you choose here will identify this level for loading later as well as creating a folder and .cry file of the same name. In the Terrain section of the dialog box, set Heightmap Resolution to 1024x1024 , and Meters Per Unit to 1. Click on OK and your New Level will begin to load. This should occur relatively fast, but will depend on your computer's specifications. You will know the level has been loaded when you see Ready in the status bar. You will also see an ocean stretching out infinitely and some terrain slightly underneath the water. Maneuver your camera so that you have a good, overall view of the map you will create, as seen in the following screenshot: (Move the mouse over the image to enlarge.) What just happened? Congratulations! You now have an empty level to mold and modify at your will. Before moving on, let's talk a little about the properties that we just set, as they are fundamental properties of the levels within CryENGINE. It is important to understand these, as depending on the type of game you are creating, you may need bigger or smaller maps, or you may not even need terrain at all. Using the right Heightmap Resolution When we created the New Level, we chose a Heightmap Resolution of 1024x1024. To explain this further, each pixel on the heightmap has a certain grey level. This pixel then gets applied to the terrain polygons, and depending on the level of grey, will move the polygon on the terrain to a certain height. This is called displacement. Heightmaps always have varying values from full white to full black, where full white is maximum displacement and full black is minimum or no displacement. The higher the resolution of the heightmap, the more the pixels that are available to represent different features on said heightmap. You can thus achieve more definition and a more accurate geometrical representation of your heightmap using higher resolutions. The settings can range from the smallest resolution of 128x128, all the way to the largest supported resolution of 8192x8192 . The following screenshot shows the difference between high resolution and low resolution heightmaps:   Scaling your level with Meters Per Unit If the Heightmap Resolution parameter is examined in terms of pixel size, then this dialog box can be viewed also as the Meters Per Pixel parameter . This means that each pixel of the heightmap will be represented by so many meters. For example, if a heightmap's resolution has 4 Meters Per Unit, then each pixel on the generated heightmap will measure to be 4 meters in length and width on the level. Even though Meters Per Unit can be used to increase the size of your level, it will decrease the fidelity of the heightmap. You will notice that attempting to smoothen out the terrain may be difficult, since there will be a wider, minimum triangle size set by this value. Keep in mind that you can adjust the unit size even after the map has been created. This is done through the terrain editor, which we will discuss shortly. Calculating the real-world size of the terrain The expected size of the terrain can easily be calculated before making the map, because the equation is not so complicated. The real-world size of the terrain can be calculated as: (Heightmap Resolution) x Meters Per Unit = Final Terrain Dimensions. For example: (128x128) x 2m = 256x256m (512x512) x 8m = 4096x4096m (1024x1024) x 2m = 2048x2048m Using or not using terrain In most cases, levels in CryENGINE will use some amount of the terrain. The terrain itself is a highly optimized system that has levels of dynamic tessellation, which adjusts the density of polygons depending on the distance from the camera to the player. Dynamic tessellation is used to make the more defined areas of the terrain closer to the camera and the less defined ones further away, as the amount of terrain polygons on the screen will have a significant impact on the performance of the level. In some cases, however, the terrain can be expensive in terms of performance, and if the game is made in an environment like space or interior corridors and rooms, then it might make sense to disable the terrain. Disabling the terrain in these cases will save an immense amount of memory, and speed up level loading and runtime performance. In this particular example, we will use the terrain, but should you wish to disable it, simply go to the second tab in the RollupBar (usually called the environment tab) and set the ShowTerrainSurface parameter to false , as shown in the following screenshot:   Time for action - creating your own heightmap You must have created a new map to follow this example. Having sufficiently beaten the terrain system to death through explanation, let's get on with what we are most interested in, which is creating our own heightmap to use for our game: As discussed in the previous example, you should now see a flat plane of terrain slightly submerged beneath the ocean. At the top of the Sandbox interface in the main toolbar, you will find a menu selection called Terrain; open this. The following screenshot shows the options available in the Terrain menu. As we want to adjust the terrain, we will select the Edit Terrain option. This will open the Terrain Editor window, which is shown in the following screenshot: You can zoom in and pan this window to further inspect areas within the map. Click-and-drag using the right mouse button to pan the view and use the mouse wheel to zoom in and zoom out. The Terrain Editor window has a multitude of options, which can be used to manipulate the heightmap of your level. Before we start painting anything, we should first set the maximum height of the map to something more manageable: Click on Modify. Click on Set Max Height. Set your Max Terrain Height to 256. Note that the terrain height is measured in meters.     Having now set the Max Height parameter, we are ready to paint! Using a second monitor: This is a good time to take advantage of a second monitor should you have one, as you can leave the perspective view on your primary monitor and view the changes made in the Terrain Editor on your second monitor, in real time. On the right-hand side of the Terrain Editor , you will see a rollout menu named Terrain Brush. We will first use this to flatten a section of the level. Change the Brush Settings to Flatten, and set the following values: Outside Radius = 100 Inside Radius = 100 Hardness = 1 Height = 20     NOTE: You can sample the terrain height in the Terrain Editor or the view port using the shortcut Control when the flatten brush is selected. Now paint over the top half of the map. This will flatten the entire upper half of the terrain to 20 meters in height. You will end up with the following screenshot, where the dark portion represents the terrain, and since it is relatively low compared to our max height, it will appear black: Note that, by default, the water is set to a height of 16 meters. Since we flattened our terrain to a height of 20 meters, we have a 4-meter difference from the terrain to the water in the center of the map. In the perspective viewport, this will look like a steep cliff going into the water. At the location where the terrain meets the water, it would make sense to turn this into a beach, as it's the most natural way to combine terrain and water. To do this, we will smoothen the hard edge of the terrain along the water. As this is to become our beach area, let's now use the smooth tools to make it passable by the player: Change the Type of brush to Smooth and set the following parameters: Outside Radius = 50 Hardness = 1 I find it significantly easier to gauge the effects of the smooth brush in the perspective viewport. Paint the southern edge of the terrain, which will become our beach. It might be difficult to view the effects of the smooth brush simply in the terrain editor, so I recommend using the perspective viewport to paint your beach. Now that we have what will be our beach, let's sculpt some background terrain. Select the Rise/Lower brush and set the following parameters: Outside Radius = 75 Inside Radius = 50 Hardness = 0.8 Height = 1 Before painting, set the Noise Settings for the brush; to do so, check Enable Noise to true. Also set: Scale = 5 Frequency = 25 Paint the outer edges of the terrain while keeping an eye on the perspective viewport at the actual height of the mountain type structure that this creates. You can see the results in the Terrain Editor and perspective view, as seen in the following screenshots: It is a good time to use the shortcut to switch to smooth brush while painting the terrain. While in perspective view, switch to the smooth brush using the Shift shortcut. A good technique is to use the Rise/Lower brush and only click a few times, and then use Shift to switch to the smooth brush and do this multiple times on the same area. This will give you some nice terrain variation, which will serve us nicely when we go to texture it. Don't forget the player's perspective: Remember to switch to game mode periodically to inspect your terrain from the players level. It is often the case that we get caught up in the appearance of a map by looking at it from our point of view while building it, rather than from the point of view of the player, which is paramount for our game to be enjoyable to anyone playing it. Save this map as Book_Example_1_no_color.cry.

  Mastering phpMyAdmin 3.3.x for Effective MySQL Management A complete guide to get started with phpMyAdmin 3.3 and master its features The best introduction to phpMyAdmin available Written by the project leader of phpMyAdmin, and improved over several editions A step-by-step tutorial for manipulating data with phpMyAdmin Learn to do things with your MySQL database and phpMyAdmin that you didn't know were possible! Managing users and their privileges The Privileges subpage (visible only if we are logged in as a privileged user) contains dialogs to manage MySQL user accounts. It also contains dialogs to manage privileges on the global, database, and table levels. This subpage is hierarchical. For example, when editing a user's privileges, we can see the global privileges as well as the database-specific privileges. We can then go deeper to see the table-specific privileges for this database-user combination. The user overview The first page displayed when we enter the Privileges subpage is called User verview. This shows all user accounts and a summary of their global privileges, as shown in the next screenshot: From this page, we can: Edit a user's privileges, via the Edit link for this user Use the checkboxes to remove users, via the Remove selected users dialog Access the page when the Add a new User dialog is available The displayed users' list has columns with the following characteristics: Privileges reload At the bottom of User Overview, the following message is displayed: Note: phpMyAdmin gets the users' privileges directly from MySQL's privilege tables. The content of these tables may differ from the privileges the server uses, if they have been changed manually. In this case, you should reload the privileges before you continue. Here, the text reload the privileges is clickable. The effective privileges (the ones against which the server bases its access decisions) are the privileges that are located in the server's memory. Privilege modifications that are made from the User overview page are made both in memory and on disk, in the mysql database. Modifications made directly to the mysql database do not have immediate effect. The reload the privileges operation reads the privileges from the database and makes them effective in memory. Adding a user The Add a new User link opens a dialog for user account creation. First, we see the panel where we'll describe the account itself: The second part of the Add a new User dialog is where we'll specify the user's global privileges, which apply to the server as a whole. Entering the username The User name menu offers two choices. Firstly, we can choose Use text field and enter a username in the box, or we can choose Any user to create an anonymous user (the blank user). Let's choose Use text field and enter bill. Assigning a host value By default, this menu is set to Any host, with % as the host value. The Local choice means "localhost". The Use host table choice (which creates a blank value in the host field) means to look in the mysql.hosts table for database-specific privileges. Choosing Use text field allows us to enter the exact host value we want. Let's choose Local. Setting passwords Even though it's possible to create a user without a password (by selecting the No password option), it's best to have a password. We have to enter it twice (as we cannot see what is entered) to confirm the intended password. A secure password should have more than eight characters, and should contain a mixture of uppercase and lowercase characters, digits, and special characters. Therefore, it's recommended to have phpMyAdmin generate a password—this is possible in JavaScript-enabled browsers. In the Generate Password dialog, clicking on Generate enters a random password (in clear text) on the screen and fills the Password and Re-type input fields with the generated password. At this point, we should note the password so that we can pass it on to the user. Understanding rights for database creation A frequent convention is to assign a user the rights to a database having the same name as this user. To accomplish this, the Database for user section offers the checkbox Create database with same name and grant all privileges. Selecting this checkbox automates the process by creating both the database (if it does not already exist) and the corresponding rights. Please note that, with this method, each user would be limited to one database (user bill, database bill). Another possibility is to allow users to create databases that have the same prefix as their usernames. Therefore, the other choice, Grant all privileges on wildcard name (username_%), performs this function by assigning a wildcard privilege. With this in place, user bill could create the databases bill_test, bill_2, bill_payroll, and so on; phpMyAdmin does not pre-create the databases in this case. Assigning global privileges Global privileges determine the user's access to all databases. Hence, these are sometimes known as "superuser privileges". A normal user should not have any of these privileges unless there is a good reason for this. Of course, if we are really creating a superuser, we will select every global privilege that he or she needs. These privileges are further divided into Data, Structure, and Administration groups. In our example, bill will not have any global privileges. Limiting the resources used We can limit the resources used by this user on this server (for example, the maximum queries per hour). Zero means no limit. We will not impose any resource limits on bill. The following screenshot shows the status of the screen just before hitting Go to create this user's definition (with the remaining fields being set to default): Editing a user profile The page used to edit a user's profile appears after a user's creation, or whenever we click on Edit for a user in the User overview page. There are four sections on this page, each with its own Go button. Hence, each section is operated independently and has a distinct purpose. Editing privileges The section for editing the user's privileges has the same look as the Add a new User dialog, and is used to view and change global privileges. Assigning database-specific privileges In this section, we define the databases to which our user has access, and his or her exact privileges on these databases. As shown in the previous screenshot, we see None because we haven't defined any privileges yet. There are two ways of defining database privileges. First, we can choose one of the existing databases from the drop-down menu: This assigns privileges only for the chosen database. We can also choose Use text field and enter a database name. We could enter a non-existent database name, so that the user can create it later (provided that we give him or her the CREATE privilege in the next panel). We can also use special characters, such as the underscore and the percent sign, for wildcards. For example, entering bill here would enable him to create a bill database, and entering bill% would enable him to create a database with any name that starts with bill. For our example, we will enter bill and then click on Go. The next screen is used to set bill's privileges on the bill database, and create table-specific privileges. To learn more about the meaning of a specific privilege, we can move the mouse over a privilege name (which is always in English), and an explanation about this privilege appears in the current language. We give SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, INDEX, and DROP privileges to bill on this database. We then click on Go. After the privileges have been assigned, the interface stays at the same place, so that we can refine these privileges further. We cannot assign table-specific privileges for the moment, as the database does not yet exist. To go back to the general privileges page of bill, click on the 'bill'@'localhost' title. This brings us back to the following, familiar page, except for a change in one section: We see the existing privileges (which we can Edit or Revoke) on the bill database for user bill, and we can add privileges for bill on another database. We can also see that bill has no table-specific privileges on the bill database. Changing the password The Change password dialog is part of the Edit user page, and we can use it either to change bill's password or to remove it. Removing the password will enable bill to login without a password. The dialog offers a choice of password hashing options, and it's recommended to keep the default of MySQL 4.1+ hashing. For more details about hashing, please visit http://dev.mysql.com/doc/refman/5.1/en/password-hashing.html

  Blender 2.5 Lighting and Rendering Bring your 3D world to life with lighting, compositing, and rendering Render spectacular scenes with realistic lighting in any 3D application using interior and exterior lighting techniques Give an amazing look to 3D scenes by applying light rigs and shadow effects Apply color effects to your scene by changing the World and Lamp color values A step-by-step guide with practical examples that help add dimensionality to your scene        Getting the right files Before we get started, we need a scene to work with. There are three scenes provided for our use—an outdoor scene, an indoor scene, and a hybrid scene that incorporates elements that are found both inside as well as outside. All these files can be downloaded from http://www.cgshark.com/lightingand-rendering/ The file we are going to use for this scene is called exterior.blend. This scene contains a tricycle, which we will light as if it were a product being promoted for a company. To download the files for this tutorial, visit http://www.cgshark.com/lighting-and-rendering/ and select exterior.blend. Blender render settings In computer graphics, a two-dimensional image is created from three-dimensional data through a computational process known as rendering. It's important to understand how to customize Blender's internal renderer settings to produce a final result that's optimized for our project, be it a single image or a full-length film. With the settings Blender provides us, we can set frame rates for animation, image quality, image resolution, and many other essential parts needed to produce that optimized final result. The Scene menu We can access these render settings through the Scene menu. Here, we can adjust a myriad of settings. For the sake of these projects, we are only going to be concerned with: Which window Blender will render our image in How render layers are set up Image dimensions Output location and file type Render settings The first settings we see when we look at the Scene menu are the Render settings. Here, we can tell Blender to render the current frame or an animation using the render buttons. We can also choose what type of window we want Blender to render our image in using the Display options. The first option (and the one chosen by default) is Full Screen. This renders our image in a window that overlaps the three-dimensional window in our scene. To restore the three-dimensional view, select the Back to Previous button at the top of the window. The next option is the Image Editor that Blender uses both for rendering as well as UV editing. This is especially useful when using the Compositor, allowing us to see our result alongside our composite node setup. By default, Blender replaces the three-dimensional window with the Image Editor. The last option is the option that Blender has used, by default, since day one—New Window. This means that Blender will render the image in a newly created window, separate from the rest of the program's interface. For the sake of these projects, we're going to keep this setting at the default setting—Full Screen. Dimensions settings These are some of the most important settings that we can set when dealing with optimizing our project output. We can set the image size, frame rate, frame range, and aspect ratio of our render. Luckily for us, Blender provides us with preset render settings, common in the film industry: HDTV 1080P HDTV 720P TV NTSC TV PAL TV PAL 16:9 Because we want to keep our render times relatively low for our projects, we're going to set our preset dimensions to TV NTSC, which results in an image 720 pixels wide by 480 pixels high. If you're interested in learning more about how the other formats behave, feel free to visit http://en.wikipedia.org/wiki/Display_resolution. Output settings These settings are an important factor when determining how we want our final product to be viewed. Blender provides us with numerous image and video types to choose from. When rendering an animation or image sequence, it's always easier to manually set the folder we want Blender to save to. We can tell Blender where we want it to save by establishing the path in the output settings. By default on Macintosh, Blender saves to the /tmp/ folder. Now that we understand how Blender's renderer works, we can start working with our scene! Establishing a workflow The key to constantly producing high-quality work is to establish a well-tested and efficient workflow. Everybody's workflow is different, but we are going to follow this series of steps: Evaluate what the scene we are lighting will require. Plan how we want to lay out the lamps in our scene. Set lamp positions, intensities, colors, and shadows, if applicable. Add materials and textures. Tweak until we're satisfied. Evaluating our scene Before we even begin to approach a computer, we need to think about our scene from a conceptual perspective. This is important, because knowing everything about our scene and the story that's taking place will help us produce a more realistic result. To help kick start this process, we can ask ourselves a series of questions that will get us thinking about what's happening in our scene. These questions can pertain to an entire array of possibilities and conditions, including: Weather What is the weather like on this particular day? What was it like the day before or the day after? Is it cloudy, sunny, or overcast? Did it rain or snow? Source of light Where is the light coming from? Is it in front of, to the side, or even behind the object? Remember, light is reflected and refracted until all energy is absorbed; this not only affects the color of the light, but the quality as well. Do we need to add additional light sources to simulate this effect? Scale of light sources What is the scale of our light sources in relation to our three-dimensional scene? Believe it or not, this factor carries a lot of weight when it comes to the quality of the final render. If any lights feel out of place, it could potentially affect the believability of the final product. The goal of these questions is to prove to ourselves that the scene we're lighting has the potential to exist in real life. It's much harder, if not impossible, to light a scene if we don't know how it could possibly act in the real world. Let's take a look at these questions. What is the weather like? In our case, we're not concerned with anything too challenging, weather wise. The goal of this tutorial is to depict our tricycle in an environment that reflects the effects of a sunny, cloudless day. To achieve this, we are going to use lights with blue and yellow hues for simulating the effect the sun and sky will have on our tricycle. What are the sources of our light and where are they coming from in relation to our scene? In a real situation, the sun would provide most of the light, so we'll need a key light that simulates how the sun works. In our case, we can use a Sun lamp. The key to positioning light sources within a three-dimensional scene is to find a compromise between achieving the desired mood of the image and effectively illuminating the object being presented. What is the scale of our light sources? The sun is rather large, but because of the nature of the Sun lamp in Blender, we don't have to worry about the scale of the lamp in our three-dimensional scene. Sometimes—more commonly when working with indoor scenes, such as the scene we'll approach later—certain light sources need to be of certain sizes in relation to our scene, otherwise the final result will feel unnatural. Although we will be using a realistic approach to materials, textures, and lighting, we are going to present this scene as a product visualization. This means that we won't explicitly show a ground plane, allowing the viewer to focus on the product being presented, in this case, our tricycle.

  Scribus 1.3.5: Beginner's Guide Create optimum page layouts for your documents using productive tools of Scribus. Master desktop publishing with Scribus Create professional-looking documents with ease Enhance the readability of your documents using powerful layout tools of Scribus Packed with interesting examples and screenshots that show you the most important Scribus tools to create and publish your documents. Creating a new layout Creating a layout in Scribus means dealing with the New Document window. It is not a complex window but be aware that many things you'll set here will be considered definitive. If these settings look simple or evident, you should consider all these settings as important. Some of them like the page size mean that you already have an idea of the final document, or atleast that you've already made some choices that won't change after it is created. Of course, Scribus will let you change this later if you change your mind, but many things you will have done in the meantime will simply have to be done again. Time for action – setting page size and paper size and margins This window is the first that opens when you launch Scribus or when you go to the File | New menu. It contains several options that need to be set. First among these options will certainly be the page size. In our case, people usually use 54x85mm (USA: 51×89mm). When you type the measurements in the Width and Height fields, the Size option, which contains the common size presets, is automatically switched to Custom. If you want to use a different system unit, just change the Default Unit value placed below. Usually, we prefer Millimeters (mm), which is quite precise without having too many significant decimals. Then, you can set the margin for your document. Professional printers are very different from desktop printers as they can print without margins. In fact, consider margins as helpers to place objects. For a small document like a business card, having small 4mm margins will be good. What just happened? Some common page sizes are: the series (the ISO standards biggest starting with A0 841x1189, that is 1m², and halving at each half step), the US formats, especially letter (216x279mm), legal (216x356mm), and tabloid (approximately 279x432mm, 11x17in), commonly used in the UK for newspapers. The best business card size When choosing the size for the business card, you'll consider the existing size often used. Is ISO 54x85.6mm better than the US 2x3.5in, or the European 55x85mm, or the Australian 55x90mm, when only a few millimeters divide them? Best is certainly to match the most commonly-used size in your country. Remember one thing: a business card must have to be easily stored and sorted. Grabbing an uncommon format can just lead to the fact that no one will be able to put your card in their wallet. Presets will be useful if you want to print locally, but don't forget that your print company crops the paper to the size you want. So don't mind being creative and do some testing. For example, you might print on an A3 size paper for your final document or in an A3+ real printing size so that you'll be able to use bleeds, as we'll explain in the following sections. Here we're talking about the page size and not the paper size, which can be double if the Document Layout is set to any option but Single Page. For all the folded documents, the page size differs from the paper size—keep that in mind. For now choose 54x85.6 in landscape: just set 54 as the height or change the orientation button if you haven't. The other setting that might interest you is the margin . In Scribus, consider the margin as a helper. In fact nobody in the professional print process will need margins. It is useful for desktop printers, which can't print up to the sheet border. As our example is much smaller than the usual paper size, we won't have any trouble with it. Scribus has some presets for margins that are available only when a layout other than Single Page is selected. For our model, 4mm to each side will be fine. If you want to set all the fields at once, just click on the chain button at the right-hand side of the margin fields. But actually, we can consider that we won't have much to write and that it would be nice if our margins could help position the text. So let's define the margins as follows: Left: 10mm Right: 40mm Top: 30mm Bottom: 2mm Choosing a layout We've already talked about this option several times but here we are again. What kind of layout would you choose? Single page will simulate what you might have in a text processor. You can have as many pages as you want but it will be printed page after page. You'll get its result when printing with your desktop printer: Double-sided will be the option you'll use when you'll need a folded document. This is useful for magazines, newsletters, books, or such documents. In this layout, the reader will see two pages side by side at once, and you can easily manage elements that will overlap both pages. The fold will be in the exact middle. Usually, unless you have a small document size like A5 or smaller, this layout is intended to be printed by a professional. 3-Fold and 4-Fold are more for commercial little brochures. Usually, you won't use it in Scribus and will prefer a Single Page layout that you'll divide later into three or four parts. Why? Because with the folded out, Scribus will consider each "fold" as a page and will print each of them on a separate sheet—a bit tricky. You can see that for a business card, where no fold is needed, the Single Page layout will be our choice. (Move the mouse over the image to enlarge.) For the moment we won't need other options, so you can click on OK. You'll get a white rectangle on a greyish workspace. The red outline is the selection indicator for the selected page. It shows the borders of the page. The blue rectangle shows where the margins are placed. Save the document as often as possible "Save the document as often as possible"—this is the first commandment of a software user, but in Scribus this is much more important for several reasons: First of all, apologies, Scribus is a very nice piece of software but still not perfect (but which one is?). It can crash sometimes, slightly more than you'd wish, and never at a time you would expect or appreciate. Saving often will help you save a lot of time doing again what you've already achieved during the day. The Scribus undo system acts on layout options but not on text manipulations. Saving often can be helpful if you make mistakes that you can't undo. In Scribus, we will use File | Save As (or Ctrl + Shift + S) to set the document name and format. It's very simple because you have no other choice than Scribus Documents *.sla. In the list, you will see sla.gz that will be used when the Compress File checkbox will be selected. Usually, a Scribus file is not that large in size and there is no real need to compress it. Of course, if the file already exists, Scribus asks whether you want to overwrite the previous one. Scribus file version Each Scribus release has enhanced the file format to be able to store the new possibilities in the file. But when saving, you cannot choose a version: Scribus will always use the current one. Every document can be opened in future Scribus releases but not in the older ones. So be careful when you need to send the file to someone or else when you're working on several computers. Once you've used Saved As, you'll just have to simply save (File | Save) or more magically use Ctrl + S, and the modifications will automatically be added to the saved document. The extra Save as Template menu will store the actual file in a special Scribus folder. When you want to create a new document with the same global aspect, you can go to the New from Template menu and grab it from the list. There are some default templates available here, but yours might be better. Saving as a template might not be the usual saving process; this is done at the end when the basics of your layout have been made. Saving as template must happen only once for a template. So we'll use it at the end of our tutorial. Basic frames for text and images The biggest part of a design job is adding frames, setting their visual aspect, and importing content into them. In our business card we'll need a logo, name, and other information. You may add a photo. Time for action – adding the logo They are several types of graphic elements in a layout. The logo is of course one of the most important. Generally, we prefer using vector logos in SVG or EPS. Let's import a logo. In the File menu choose File | Import | Get Vector File. The cursor has now been changed, and you can click on the page where you want to place the logo. Try to click at the upper-left corner of the margins. It will certainly not be correctly placed and the logo may be too big. We'll soon see how to change it. A warning will appear and inform you that some SVG features will not be supported. There is no option other than clicking on OK, and everything should be good. What just happened? The logo is the master piece of the card. It helps recognize the origin of the contact. In some ways, it is the most important recognition for a company. Usually, a logo is the only graphical element on the card. It can be put anywhere you want, but generally the upper left-hand side corner is the place of choice.  

Dive deeper into the world of AI innovation and stay ahead of the AI curve! Subscribe to our AI_Distilled newsletter for the latest insights. Don't miss out – sign up today!This article is an excerpt from the book, Pretrain Vision and Large Language Models in Python, by Emily Webber. Master the art of training vision and large language models with conceptual fundaments and industry-expert guidance. Learn about AWS services and design patterns, with relevant coding examplesIntroductionIn this article, we’ll explore trends in foundation model application development, like using LangChain to build interactive dialogue applications, along with techniques like retrieval augmented generation to reduce LLM hallucination. We’ll explore ways to use generative models to solve classification tasks, human-centered design, and other generative modalities like code, music, product documentation, powerpoints, and more! We’ll talk through AWS offerings like SageMaker JumpStart Foundation Models, Amazon Bedrock, Amazon Titan, and Amazon Code Whisperer.In particular, we’ll dive into the following topics:Techniques for building applications for LLMsGenerative modalities outside of vision and languageAWS offerings in foundation modelsTechniques for building applications for LLMsNow that you’ve learned about foundation models, and especially large language models, let’s talk through a few key ways you can use them to build applications. One of the most significant takeaways of the ChatGPT moment in December 2022 is that customers clearly love for their chat to be knowledgeable about every moment in the conversation, remember topics mentioned earlier, and encompassing all the twists and turns of dialogue. Said another way, beyond generic question answering, there’s a clear consumer preference for a chat to be chained. Let’s take a look at an example in the following screenshot:Figure 15.1 – Chaining questions for chat applicationsThe key difference between the left - and the right-hand side of Figure 15.1 is that on the left-hand side, the answers are discontinuous. That means the model simply sees each question as a single entity before providing its response. On the right-hand side, however, the answers are continuous. That means the entire dialogue is provided to the model, with the newest question at the bottom. This helps to ensure the continuity of responses, with the model more capable of maintaining the context.How can you set this up yourself? Well, on the one hand, what I’ve just described isn’t terribly difficult. Imagine just reading from your HTML page, packing in all of that call and response data into the prompt, and siphoning out the response to return it to your end user. If you don’t want to build it yourself, however, you can just use a few great open-source options!Building interactive dialogue apps with open-source stacksIf you haven’t seen it before, let me quickly introduce you to LangChain. Available for free on GitHub here: https://github.com/hwchase17/langchain, LangChain is an open-source toolkit built by Harrison Chase and more than 600 other contributors. It provides functionality similar to the famous ChatGPT by pointing to OpenAI’s API, or any other foundation model, but letting you as the developer and data scientist create your own frontend and customer experience.Decoupling the application from the model is a smart move; in the last few months alone the world has seen nothing short of hundreds of new large language models come online, with teams around the world actively developing more. When your application interacts with the model via a single API call, then you can more easily move from one model to the next as the licensing, pricing, and capabilities upgrade over time. This is a big plus for you!Another interesting open-source technology here is Haystack (26).  Developed by the German start-up, Deepset, Haystack is a useful tool for, well, finding a needle in a haystack. Specifically, they operate like an interface for you to bring your own LLMs into expansive question/answering scenarios. This was their original area of expertise, and since then have expanded quite a bit!At AWS, we have an open-source template for building applications with LangChain on AWS. It’s available on GitHub here: https://github.com/3coins/langchain-aws-template.In the following diagram, you can see a quick representation of the architecture:While this can point to any front end, we provide an example template you can use to get off the ground for your app. You can also easily point to any custom model, whether it’s on a SageMaker endpoint or in the new AWS service, Bedrock! More on that a bit later in this chapter. As you can see in the previous image, in this template you can easily run a UI anywhere that interacts with the cloud. Let’s take a look at all of the steps.:1.      First, the UI hits the API gateway.2.      Second, credentials are retrieved via IAM.3.      Third, the service is invoked via Lambda.4.      Fourth, the model credentials are retrieved via Secrets Manager.5.      Fift h, your model is invoked through either an API call to a serverless model SDK, or a custom model you’ve trained that is hosted on a SageMaker endpoint is invoked.6.      Sixth, look up the relevant conversation history in DynamoDB to ensure your answer is accurate.How does this chat interface ensure it’s not hallucinating answers? How does it point to a set of data stored in a database? Through retrieval augmented generation (RAG), which we will cover next.Using RAG to ensure high accuracy in LLM applicationsAs explained in the original 2020 (1) paper, RAG is a way to retrieve documents relevant to a given query. Imagine your chat application takes in a question about a specific item in your database, such as one of your products. Rather than having the model make up the answer, you’d be better off retrieving the right document from your database and simply using the LLM to stylize the response. That’s where RAG is so powerful; you can use it to ensure the accuracy of your generated answers stays high, while keeping the customer experience consistent in both style and tone. Let’s take a closer look:Figure 15.3 – RAGFirst, a question comes in from the left-hand side. In the top left , you can see a simple question, Defi ne “middle ear”. This is processed by a query encoder, which is simply a language model producing an embedding of the query. This embedding is then applied to the index of a database, with many candidate algorithms in use here: K Nearest Neighbors, Maximum Inner Product Search (MIPS), and others. Once you’ve retrieved a set of similar documents, you can feed the best ones into the generator, the final model on the right-hand side. This takes the input documents and returns a simple answer to the question. Here, the answer is The middle ear includes the tympanic cavity and the three ossicles.Interestingly, however, the LLM here doesn’t really define what the middle ear is. It’s actually answering the question, “what objects are contained within the middle ear?” Arguably, any definition of the middle ear would include its purpose, notably serving as a buffer between your ear canal and your inner ear, which helps you keep your balance and lets you hear. So, this would be a good candidate for expert reinforcement learning with human feedback, or RLHF, optimization.As shown in Figure 15.3, this entire RAG system is tunable. That means you can and should fine-tune the encoder and decoder aspects of the architecture to dial in model performance based on your datasets and query types. Another way to classify documents, as we’ll see, is generation!Is generation the new classification?As we learned in Chapter 13, Prompt Engineering, there are many ways you can push your language model to output the type of response you are looking for. One of these ways is actually to have it classify what it sees in the text! Here is a simple diagram to illustrate this concept:Figure 15.4 – Using generation in place of classificationAs you can see in the diagram, with the traditional classification you train the model ahead of time to perform one task: classification. This model may do well on classification, but it won’t be able to handle new tasks at all. This key drawback is one of the main reasons why foundation models, and especially large language models, are now so popular: they are extremely flexible and can handle many different tasks without needing to be retrained.On the right-hand side of Figure 15.4, you can see we’re using the same text as the starting point, but instead of passing it to an encoder-based text model, we’re passing it to a decoder-based model and simply adding the instruction to classify this sentence into positive or negative sentiment. You could just as easily say, “tell me more about how this customer really feels,” or “how optimistic is this home buyer?” or “help this homebuyer find a different house that meets their needs.” Arguably each of those three instructions is slightly different, veering away from pure classification and into more general application development or customer experience. Expect to see more of this over time! Let’s look at one more key technique for building applications with LLMs: keeping humans in the loop.Human-centered design for building applications with LLMsWe touched on this topic previously, in Chapter 2, Dataset Preparation: Part One, Chapter 10, FineTuning and Evaluating, Chapter 11, Detecting, Mitigating, and Monitoring Bias, and Chapter 14, MLOps for Vision and Language. Let me say this yet again; I believe that human labeling will become even more of a competitive advantage that companies can provide. Why? Building LLMs is now incredibly competitive; you have both the open source and proprietary sides actively competing for your business. Open source options are from the likes of Hugging Face and Stability, while proprietary offerings are from AI21, Anthropic, and OpenAI. The differences between these options are questionable; you can look up the latest models at the top of the leaderboard from Stanford’s HELM (2), which incidentally falls under their human-centered AI initiative. With enough fine-tuning and customization, you should generally be able to meet performance.What then determines the best LLM applications, if it’s not the foundation model? Obviously, the end-to-end customer experience is critical, and will always remain so. Consumer preferences wax and wane over time, but a few tenets remain for general technology: speed, simplicity, flexibility, and low cost. With foundation models we can clearly see that customers prefer explainability and models they can trust. This means that application designers and developers should grapple with these long-term consumer preferences, picking solutions and systems that maximize them. As you may have guessed, that alone is no small task.Beyond the core skill of designing and building successful applications, what else can we do to stay competitive in this brave new world of LLMs? I would argue that amounts to customizing your data. Focus on making your data and your datasets unique: singular in purpose, breadth, depth, and completeness. Lean into labeling your data with the best resources you can, and keep that a core part of your entire application workflow. This brings you to continuous learning, or the ability of the model to constantly get better and better based on signals from your end users.Next, let’s take a look at upcoming generative modalities.Other generative modalitiesSince the 2022 ChatGPT moment, most of the technical world has been fascinated by the proposition of generating novel content. While this was always somewhat interesting, the meeting of high-performance foundation models with an abundance of media euphoria over the capabilities, combined with a post-pandemic community with an extremely intense fear of missing out, has led us to the perfect storm of a global fixation on generative AI.Is this a good thing? Honestly, I’m happy to finally see the shift ; I’ve been working on generating content with AI/ML models in some fashion since at least 2019, and as a writer and creative person myself, I’ve always thought this was the most interesting part of machine learning. I was very impressed by David Foster’s book (3) on the topic. He’s just published an updated version of this to include the latest foundation models and methods! Let’s quickly recap some other types of modalities that are common in generative AI applications today.Generating code should be no surprise to most of you; its core similarities to language generation make it a perfect candidate! Fine-tuning an LLM to spit out code in your language of choice is pretty easy; here’s my 2019 project (4) doing exactly that with the SageMaker example notebooks! Is the code great? Absolutely not, but fortunately, LLMs have come a long way since then. Many modern code-generating models are excellent, and thanks to a collaboration between Hugging Face and ServiceNow we have an open-source model to use! This is called StarCoder and is available for free on HuggingFace right here: https://huggingface.co/bigcode/starcoder.What I love about using an open-source LLM for code generation is that you can customize it! This means you can point to your own private code repositories, tokenize the data, update the model, and immediately train this LLM to generate code in the style of your organization! At the organizational level, you might even do some continued pretraining on an open-source LLM for code generation on your own repositories to speed up all of your developers. We’ll take a look at more ways you can useLLMs to write your own code faster in the next section when we focus on AWS offerings, especially Amazon Code Whisperer. (27)The rest of the preceding content can all be great candidates for your own generative AI projects. Truly, just as we saw general machine learning moving from the science lab into the foundation of most businesses and projects, it’s likely that generative capabilities in some fashion will do the same.Does that mean engineering roles will be eliminated? Honestly, I doubt it. Just as the rise of great search engines didn’t eliminate software engineering roles but made them more fun and doable for a lot of people, I’m expecting generative capabilities to do the same. They are great at searching many possibilities and quickly finding great options, but it’s still up to you to know the ins and outs of your consumers, your product, and your design. Models aren’t great at critical thinking, but they are good at coming up with ideas and finding shortcomings, at least in words.Now that we’ve looked at other generative modalities at a very high level, let’s learn about AWS offerings for foundation models!AWS offerings in foundation modelsOn AWS, as you’ve seen throughout the book, you have literally hundreds of ways to optimize your foundation model development and operationalization. Let’s now look at a few ways AWS is explicitly investing to improve the customer experience in this domain:SageMaker JumpStart Foundation Model Hub: Announced in preview at re: Invent 2022, this is an option for pointing to foundation models nicely packaged in the SageMaker environment. This includes both open-source models such as BLOOM and Flan-T5 from Hugging Face, and proprietary models such as AI21 Jurassic. A list of all the foundation models is available here (5). To date, we have nearly 20 foundation models, all available for hosting in your own secure environments. Any data you use to interact with or fine-tune models on the Foundation Model Hub is not shared with providers. You can also optimize costs by selecting the instances yourself. We have tens of example notebooks pointing to these models for training and hosting across a wide variety of use cases available here (6) and elsewhere. For more information about the data the models were trained on, you can read about that in the playground directly.Amazon Bedrock: If you have been watching AWS news closely in early 2023, you may have noticed a new service we announced for foundation models: Amazon Bedrock! As discussed in this blog post (7) by Swami Sivasubramanian, Bedrock is a service that lets you interact with a variety of foundation models through a serverless interface that stays secure. Said another way, Bedrock provides a point of entry for multiple foundation models, letting you get the best of all possible providers. This includes AI start-ups such as AI21, Anthropic, and Stability. Interacting with Bedrock means invoking a serverless experience, saving you from dealing with the lower-level infrastructure. You can also fine-tune your models with Bedrock!Amazon Titan: Another model that will be available through Bedrock is Titan, a new large language model that’s fully trained and managed by Amazon! This means we handle the training data, optimizations, tuning, debiasing, and all enhancements for getting you results with large language models. Titan will also be available for fine-tuning.Amazon Code Whisperer: As you may have seen, Code Whisperer is an AWS service announced in 2022 and made generally available in 2023. Interestingly it seems to tightly couple with a given development environment, taking the entire context of the script you are writing and generating recommendations based on this. You can write pseudo-code, markdown, or other function starts, and using keyboard shortcuts invoke the model. This will send you a variety of options based on the context of your script, letting you ultimately select the script that makes the most sense for you! Happily, this is now supported for both Jupyter notebooks and SageMaker Studio; you can read more about these initiatives from AWS Sr Principal Technologist Brain Granger, co-founder of Project Jupyter. Here’s Brian’s blog post on the topic: https://aws.amazon.com/blogs/machine-learning/announcing-new-jupyter-contributions-by-aws-to-democratize-generative-ai-and-scale-ml-workloads/ Pro tip: Code Whisperer is free to individuals! Close readers of Swami’s blog post above will also notice updates to our latest ML infrastructure, like the second edition of the inferentia chip, inf2, and a trainium instance with more bandwidth, trn1n.Close readers of Swami’s blog post will also notice updates to our latest ML infrastructure, such as the second edition of the inferentia chip, inf2, and a Trainium instance with more bandwidth, trn1n. We also released our code generation service, CodeWhisperer, at no cost to you!ConclusionIn summary, the field of pretraining foundation models is filled with innovation. We have exciting advancements like LangChain and AWS's state-of-the-art solutions such as Amazon Bedrock and Titan, opening up vast possibilities in AI development. Open-source tools empower developers, and the focus on human-centered design remains crucial. As we embrace continuous learning and explore new generative methods, we anticipate significant progress in content creation and software development. By emphasizing customization, innovation, and responsiveness to user preferences, we stand on the cusp of fully unleashing the potential of foundation models, reshaping the landscape of AI applications. Keep an eye out for the thrilling journey ahead in the realm of AI.Author BioEmily Webber is a Principal Machine Learning Specialist Solutions Architect at Amazon Web Services. She has assisted hundreds of customers on their journey to ML in the cloud, specializing in distributed training for large language and vision models. She mentors Machine Learning Solution Architects, authors countless feature designs for SageMaker and AWS, and guides the Amazon SageMaker product and engineering teams on best practices in regards around machine learning and customers. Emily is widely known in the AWS community for a 16-video YouTube series featuring SageMaker with 160,000 views, plus a Keynote at O’Reilly AI London 2019 on a novel reinforcement learning approach she developed for public policy.

(For more resources related to this topic, see here.) Introducing cloud Before embarking on a journey to understand and appreciate CloudStack, let's revisit the basic concepts of cloud computing and how CloudStack can help us in achieving our private, public, or hybrid cloud objectives. Let's start this article with a plain and simple definition of cloud. Cloud is a shared multi-tenant environment built on a highly efficient, highly automated, and preferably virtualized IT infrastructure where IT resources can be provisioned on demand from anywhere over a broad network, and can be metered. Virtualization is the technology that has made the enablement of these features simpler and convenient. A cloud can be deployed in various models; including private, public, community or hybrid clouds. These deployment models can be explained as follows: Private cloud: In this deployment model, the cloud infrastructure is operated solely for an organization and may exist on premise or off premise. It can be managed by the organization or a third-party cloud provider. Public cloud: In this deployment model, the cloud service is provided to the general public or a large industry group, and is owned and managed by the organization providing cloud services. Community cloud: In this deployment model, the cloud is shared by multiple organizations and is supported by a specific community that has shared concerns. It can be managed by the organization or a third party provider, and can exist on premise or off premise. Hybrid cloud: This deployment model comprises two or more types of cloud (public, private, or community) and enables data and application portability between the clouds. A cloud—be it private, public, or hybrid—has the following essential characteristics: On-demand self service Broad network access Resource pooling Rapid elasticity or expansion Measured service Shared by multiple tenants Cloud has three possible service models, which means there are three types of cloud services that can be provided. They are: Infrastructure as a service (IaaS): This type of cloud service model provides IT infrastructure resources as a service to the end users. This model provides the end users with the capability to provision processing, storage, networks, and other fundamental computing resources that the customer can use to run arbitrary software including operating systems and applications. The provider manages and controls the underlying cloud infrastructure and the user has control over the operating systems, storage and deployed applications. The user may also have some control over the networking services. Platform as a service (PaaS): In this service model, the end user is provided with a platform that is provisioned over the cloud infrastructure. The provider manages the network, operating system, or storage and the end user has control over the applications and may have control over the hosting environment of the applications. Software as a service (SaaS): This layer provides software as a service to the end users, such as providing an online calculation engine for their end users. The end users can access these software using a thin client interface such as a web browser. The end users do not manage the underlying cloud infrastructure such as network, servers, OS, storage, or even individual application capabilities but may have some control over the application configurations settings. As depicted in the preceding diagram, the top layers of cloud computing are built upon the layer below it. In this book, we will be mainly dealing with the bottom layer—Infrastructure as a service. Thus providing Infrastructure as a Service essentially means that the cloud provider assembles the building blocks for providing these services, including the computing resources hardware, networking hardware and storage hardware. These resources are exposed to the consumers through a request management system which in turn is integrated with an automated provisioning layer. The cloud system also needs to meter and bill the customer on various chargeback models. The concept of virtualization enables the provider to leverage and pool resources in a multi-tenant model. Thus, the features provided by virtualization resource pooling, combined with modern clustering infrastructure, enable efficient use IT resources to provide high availability and scalability, increase agility, optimize utilization, and provide a multi-tenancy model. One can easily get confused about the differences between the cloud and a virtualized Datacenter; well, there are many differences, such as: The cloud is the next stage after the virtualization of datacenters. It is characterized by a service layer over the virtualization layer. Instead of bare computing resources, services are built over the virtualization platforms and provided to the users. Cloud computing provides the request management layer, provisioning layer, metering and billing layers along with security controls and multi-tenancy. Cloud resources are available to consumers on an on demand model wherein the resources can be provisioned and de-provisioned on an as needed basis. Cloud providers typically have huge capacities to serve variable workloads and manage variable demand from customers. Customers can leverage the scaling capabilities provided by cloud providers to scale up or scale down the IT infrastructure needed by the application and the workload. This rapid scaling helps the customer save money by using the capacity only when it is needed. The resource provisioning in the cloud is governed by policies and rules, and the process of provisioning is automated. Metering, Chargeback, and Billing are essential governance characteristics of any cloud environment as they govern and control the usage of precious IT resources. Thus setting up a cloud is basically building capabilities to provide IT resources as a service in a well-defined manner. Services can be provided to end users in various offerings, depending upon the amount of resources each service offering provides. The amount of resources can be broken down to multiple resources such as the computing capacity, memory, storage, network bandwidth, storage IOPS, and so on. A cloud provider can provide and meter multiple service offerings for the end users to choose from. Though the cloud provider makes upfront investments in creating the cloud capacity, however from a consumer's point of view the resources are available on demand on a pay per use model. Thus the customer gets billed for consumption just like in case of electricity or telecom services that individuals use. The billing may be based on hours of compute usage, the amount of storage used, bandwidth consumed, and so on. Having understood the cloud computing model, let's look at the architecture of a typical Infrastructure as a Service cloud environment. Infrastructure layer The Infrastructure layer is the base layer and comprises of all the hardware resources upon which IT is built upon. These include computing resources, storage resources, network resources, and so on. Computing resources Virtualization is provided using a hypervisor that has various functions such as enabling the virtual machines of the hosts to interact with the hardware. The physical servers host the hypervisor layer. The physical server resources are accessed through the hypervisor. The hypervisor layer also enables access to the network and storage. There are various hypervisors on the market such as VMware, Hyper-V, XenServer, and so on. These hypervisors are responsible for making it possible for one physical server to host multiple machines, and for enabling resource pooling and multi tenancy. Storage Like the Compute capacity, we need storage which is accessible to the Compute layer. The Storage in cloud environments is pooled just like the Compute and accessed through the virtualization layer. Certain types of services just offer storage as a service where the storage can be programmatically accessed to store and retrieve objects. Pooled, virtualized storage is enabled through technologies such as Network Attached Storage (NAS) and Storage Area Network (SAN) which helps in allowing the infrastructure to allocate storage on demand that can be based on policy, that is, automated. The storage provisioning using such technologies helps in providing storage capacity on demand to users and also enables the addition or removal of capacity as per the demand. The cost of storage can be differentiated according to the different levels of performance and classes of storage. Typically, SAN is used for storage capacity in the cloud where statefulness is required. Direct-attached Storage (DAS) can be used for stateless workloads that can drive down the cost of service. The storage involved in cloud architecture can be redundant and prevent the single point of failure. There can be multiple paths for the access of disk arrays to provide redundancy in case connectivity failures. The storage arrays can also be configured in a way that there is incremental backup of the allocated storage. The storage should be configured such that health information of the storage units is updated in the system monitoring service, which ensures that the outage and its impact are quickly identified and appropriate action can be taken in order to restore it to its normal state. Networks and security Network configuration includes defining the subnets, on-demand allocation of IP addresses, and defining the network routing tables to enable the flow of data in the network. It also includes enabling high availability services such as load balancing. Whereas the security configuration aims to secure the data flowing in the network that includes isolation of data of different tenants among each other and with the management data of cloud using techniques such as network isolation and security groups. Networking in the cloud is supposed to deal with the isolation of resources between multiple tenants as well as provide tenants with the ability to create isolated components. Network isolation in the cloud can be done using various techniques of network isolation such as VLAN, VXLAN, VCDNI, STT, or other such techniques. Applications are deployed in a multi-tenant environment and consist of components that are to be kept private, such as a database server which is to be accessed only from selected web servers and any other traffic from any other source is not permitted to access it. This is enabled using network isolation, port filtering, and security groups. These services help with segmenting and protecting various layers of application deployment architecture and also allow isolation of tenants from each other. The provider can use security domains, layer 3 isolation techniques to group various virtual machines. The access to these domains can be controlled using providers' port filtering capabilities or by the usage of more stateful packet filtering by implementing context switches or firewall appliances. Using network isolation techniques such as VLAN tagging and security groups allows such configuration. Various levels of virtual switches can be configured in the cloud for providing isolation to the different networks in the cloud environment. Networking services such as NAT, gateway, VPN, Port forwarding, IPAM systems, and access control management are used in the cloud to provide various networking services and accessibility. Some of these services are explained as follows: NAT: Network address translation can be configured in the environment to allow communication of a virtual machine in private network with some other machine on some other network or on the public Internet. A NAT device allows the modification of IP address information in the headers of IP packets while they are transformed from a routing device. A machine in a private network cannot have direct access to the public network so in order for it to communicate to the Internet, the packets are sent to a routing device or a virtual machine with NAT configured which has direct access to the Internet. NAT modifies the IP packet header so that the private IP address of the machine is not visible to the external networks. IPAM System/DHCP: An IP address management system or DHCP server helps with the automatic configuration of IP addresses to the virtual machines according to the configuration of the network and the IP range allocated to it. A virtual machine provisioned in a network can be assigned an IP address as per the user or is assigned an IP address from the IPAM. IPAM stores all the available IP addresses in the network and when a new IP address is to be allocated to a device, it is taken from the available IP pool, and when a device is terminated or releases the IP address, the address is given back to the IPAM system. Identity and access management: A access control list describes the permissions of various users on different resources in the cloud. It is important to define an access control list for users in a multi-tenant environment. It helps in restricting actions that a user can perform on any resource in the cloud. A role-based access mechanism is used to assign roles to users' profile which describes the roles and permissions of users on different resources. Use of switches in cloud A switch is a LAN device that works at the data link layer (layer 2) of the OSI model and provides multiport bridge. Switches store a table of MAC addresses and ports. Let us see the various types of switches and their usage in the cloud environment: Layer 3 switches: A layer-3 switch is a special type of switch which operates at layer 3—the Network layer of the OSI model. It is a high performance device that is used for network routing. A layer-3 switch has a IP routing table for lookups and it also forms a broadcast domain. Basically, a layer-3 switch is a switch which has a router's IP routing functionality built in. A layer-3 switch is used for routing and is used for better performance over routers. The layer-3 switches are used in large networks like corporate networks instead of routers. The performance of the layer-3 switch is better than that of a router because of some hardware-level differences. It supports the same routing protocols as network routers do. The layer-3 switch is used above the layer-2 switches and can be used to configure the routing configuration and the communication between two different VLANs or different subnets. Layer 4-7 switches: These switches use the packet information up to OSI layer 7 and are also known as content switches, web-switches, or application switches. These types of switches are typically used for load balancing among a group of servers which can be performed on HTTP, HTTPS, VPN, or any TCP/IP traffic using a specific port. These switches are used in the cloud for allowing policy-based switching—to limit the different amount of traffic on specific end-user switch ports. It can also be used for prioritizing the traffic of specific applications. These switches also provide forwarding decision making like NAT services and also manages the state of individual sessions from beginning to end thus acting like firewalls. In addition, these switches are used for balancing traffic across a cluster of servers as per the configuration of the individual session information and status. Hence these types of switches are used above layer-3 switches or above a cluster of servers in the environment. They can be used to forward packets as per the configuration such as transferring the packets to a server that is supposed to handle the requests and this packet forwarding configuration is generally based on the current server loads or sticky bits that binds the session to a particular server. Layer-3 traffic isolation provides traffic isolation across layer-3 devices. It's referred to as Virtual Routing and Forwarding (VRF). It virtualizes the routing table in a layer-3 switch and has set of virtualized tables for routing. Each table has a unique set of forwarding entries. Whenever traffic enters, it is forwarded using the routing table associated with the same VRF. It enables logical isolation of traffic as it crosses a common physical network infrastructure. VRFs provide access control, path isolation, and shared services. Security groups are also an example of layer-3 isolation capabilities which restricts the traffic to the guests based on the rules defined. The rules are defined based on the port, protocol, and source/destination of the traffic. Virtual switches: The virtual switches are software program that allows one guest VM to communicate with another and is similar to the Ethernet switch explained earlier. Virtual switches provide a bridge between the virtual NICs of the guest VMs and the physical NIC of the host. Virtual switches have port groups on one side which may or may not be connected to the different subnets. There are various types of virtual switches used with various virtualization technologies such as VMware Vswitch, Xen, or Open Vswitch. VMware also provides a distributed virtual switch which spans multiple hosts. The virtual switches consists of port groups at one end and an uplink at the other. The port groups are connected to the virtual machines and the uplink is mapped to the physical NIC of the host. The virtual switches function as a virtual switch over the hypervisor layer on the host. Management layer The Management layer in a cloud computing space provides management capabilities to manage the cloud setup. It provides features and functions such as reporting, configuration for the automation of tasks, configuration of parameters for the cloud setup, patching, and monitoring of the cloud components. Automation The cloud is a highly automated environment and all tasks such as provisioning the virtual machine, allocation of resources, networking, and security are done in a self-service mode through automated systems. The automation layer in cloud management software is typically exposed through APIs. The APIs allow the creation of SDKs, scripts, and user interfaces. Orchestration The Orchestration layer is the most critical interface between the IT organization and its infrastructure, and helps in the integration of the various pieces of software in the cloud computing platform. Orchestration is used to join together various individual tasks which are executed in a specified sequence with exception handling features. Thus a provisioning task for a virtual machine may involve various commands or scripts to be executed. The orchestration engine binds these individual tasks together and creates a provisioning workflow which may involve provisioning a virtual machine, adding it to your DNS, assigning IP Addresses, adding entries in your firewall and load balancer, and so on. The orchestration engine acts as an integration engine and also provides the capabilities to run an automated workflow through various subsystems. As an example, the service request to provision cloud resources may be sent to an orchestration engine which then talks to the cloud capacity layer to determine the best host or cluster where the workload can be provisioned. As a next step, the orchestration engine chooses the component to call to provision the resources. The orchestration platform helps in easy creation of complex workflows and also provides ease of management since all integrations are handled by a specialized orchestration engine and provide loose coupling. The orchestration engine is executed in the cloud system as an asynchronous job scheduler which orchestrates the service APIs to fulfill and execute a process. Task Execution The Task execution layer is at the lower level of the management operations that are performed using the command line or any other interface. The implementation of this layer can vary as per the platform on which the execution takes place. The activity of this layer is activated by the layers above in the management layer. Service Management The Service Management layer helps in compliance and provides means to implement automation and adapts IT service management best practices as per the policies of the organization, such as the IT Infrastructure Library (ITIL). This is used to build processes to implement different types of incident resolutions and also provide change management. The self service capability in cloud environment helps in providing users with a self-service catalog which consists of various service options that the user can request and provision resources from the cloud. The service layer can be comprised of various levels of services such as basic provisioning of virtual machines with some predefined templates/configuration, or can be of an advanced level with various options for provisioning servers with configuration options as well.

IntroductionIn the realm of AI and machine learning, effective management of vast high-dimensional vector data is critical. Milvus, an open-source vector database, tackles this challenge using advanced indexing for swift similarity search and analytics, catering to AI-driven applications.Milvus operates on vectorization and quantization, converting complex raw data into streamlined high-dimensional vectors for efficient indexing and querying. Its scope spans recommendation, image recognition, natural language processing, and bioinformatics, boosting result precision and overall efficiency.Milvus impresses not just with capabilities but also design flexibility, supporting diverse backends like MinIO, Ceph, AWS S3, Google Cloud Storage, alongside etcd for metadata storage.Local Milvus deployment becomes user-friendly with Docker Compose, managing multi-container Docker apps well-suited for Milvus' distributed architecture. The guide delves into Milvus' core principles—vectorization and quantization—reshaping raw data into compact vectors for efficient querying. Its applications in recommendation, image recognition, natural language processing, and bioinformatics enhance system accuracy and efficacy.The next article details deploying Milvus locally via Docker Compose. This approach's simplicity underscores Milvus' user-centric design, delivering robust capabilities within an accessible framework. Let’s get started.Standalone Milvus with Docker ComposeSetting up a local instance of Milvus involves a multi-service architecture that consists of the Milvus server, metadata storage, and object storage server. Docker Compose provides an ideal environment to manage such a configuration in a convenient and efficient way.The Docker Compose file for deploying Milvus locally consists of three services: etcd, minio, and milvus itself. etcd provides metadata storage, minio functions as the object storage server and milvus handles vector data processing and search. By specifying service dependencies and environment variables, we can establish seamless communication between these components. milvus, etcd, and minio services are run in isolated containers, ensuring operational isolation and enhanced security.To launch the Milvus application, all you need to do is execute the Docker Compose file. Docker Compose manages the initialization sequence based on service dependencies and takes care of launching the entire stack with a single command. The next is the docker-compose.yml which specifies all of the aforementioned components:version: '3' services: etcd:    container_name: milvus-etcd    image: quay.io/coreos/etcd:v3.5.5    environment:      - ETCD_AUTO_COMPACTION_MODE=revision      - ETCD_AUTO_COMPACTION_RETENTION=1000      - ETCD_QUOTA_BACKEND_BYTES=4294967296      - ETCD_SNAPSHOT_COUNT=50000    command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls <http://0.0.0.0:2379> --data-dir /etcd minio:    container_name: milvus-minio    image: minio/minio:RELEASE.2022-03-17T06-34-49Z    environment:      MINIO_ACCESS_KEY: minioadmin      MINIO_SECRET_KEY: minioadmin    ports:      - "9001:9001"      - "9000:9000"    command: minio server /minio_data --console-address ":9001"    healthcheck:      test: ["CMD", "curl", "-f", "<http://localhost:9000/minio/health/live>"]      interval: 30s      timeout: 20s      retries: 3 milvus:    container_name: milvus-standalone    image: milvusdb/milvus:v2.3.0-beta    command: ["milvus", "run", "standalone"]    environment:      ETCD_ENDPOINTS: etcd:2379      MINIO_ADDRESS: minio:9000    ports:      - "19530:19530"      - "9091:9091"    depends_on:      - "etcd"      - "minio"After we have defined the docker-compose file we can deploy the services by first running docker compose build and then running docker compose up -d.In the next section, we'll move on to a practical example — creating sentence embeddings. This process leverages Transformer models to convert sentences into high-dimensional vectors. These embeddings capture the semantic essence of the sentences and serve as an excellent demonstration of the sort of data that can be stored and processed with Milvus.Creating sentence embeddingsCreating sentence embeddings involves a few steps: preparing your environment, importing necessary libraries, and finally, generating and processing the embeddings. We'll walk through each step in this section assuming that this code is being executed in a Python environment where the Milvus database is running.First, let’s start with the requirements.txt file:transformers==4.25.1 pymilvus==2.1.0 torch==2.0.1 protobuf==3.18.0 Now let’s import the packages. import numpy as np import torch import torch.nn.functional as F from transformers import AutoTokenizer, AutoModel from pymilvus import (    connections,    utility,    FieldSchema, CollectionSchema, DataType,    Collection, )Here, we're importing all the necessary libraries for our task. numpy and torch are used for mathematical operations and transformations, transformers is for language model-related tasks, and pymilvus is for interacting with the Milvus server.This Python code block sets up the transformer model we will be using and lists the sentences for which we will generate embeddings. We first specify a model checkpoint ("sentence-transformers/all-MiniLM-L6-v2") that will serve as our base model for sentence embeddings. We then define a list of sentences to generate embeddings for. To facilitate our task, we initialize a tokenizer and model using the model checkpoint. The tokenizer will convert our sentences into tokens suitable for the model, and the model will use these tokens to generate embeddings:# Transformer model checkpoint model_ckpt = "sentence-transformers/all-MiniLM-L6-v2" # Sentences for which we will compute embeddings sentences = [    "I took my dog for a walk",    "Today is going to rain",    "I took my cat for a walk", ] # Initialize tokenizer and model tokenizer = AutoTokenizer.from_pretrained(model_ckpt) model = AutoModel.from_pretrained(model_ckpt)Here, we define the model checkpoint that we will use to get the sentence embeddings. We then initialize a list of sentences for which we will compute embeddings. The tokenizer and model are initialized using the defined checkpoint.We've obtained token embeddings, but we need to aggregate them to obtain sentence-level embeddings. For this, we'll use a mean pooling operation. The upcoming section of the guide will define a function to accomplish this.Mean Pooling Function DefinitionThis function is used to aggregate the token embeddings into sentence embeddings. The token embeddings and the attention mask (which indicates which tokens are not padding and should be considered for pooling) are passed as inputs to this function. The function performs a weighted average of the token embeddings according to the attention mask and returns the aggregated sentence embeddings:# Mean pooling function to aggregate token embeddings into sentence embeddings def mean_pooling(model_output, attention_mask):    token_embeddings = model_output.last_hidden_state    input_mask_expanded = (        attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()    )    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(       input_mask_expanded.sum(1), min=1e-9    )This function takes the model output and the attention mask as input and returns the sentence embeddings by performing a mean pooling operation over the token embeddings. The attention mask is used to ignore the tokens corresponding to padding during the pooling operation.Generating Sentence EmbeddingsThis code snippet first tokenizes the sentences, padding and truncating them as necessary. We then use the transformer model to generate token embeddings. These token embeddings are pooled using the previously defined mean pooling function to create sentence embeddings. The embeddings are normalized to ensure consistency and finally transformed into Python lists to make them compatible with Milvus:# Tokenize the sentences and compute their embeddings encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors="pt") with torch.no_grad():    model_output = model(**encoded_input) sentence_embeddings = mean_pooling(model_output, encoded_input["attention_mask"]) # Normalize the embeddings sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=1) # Convert the sentence embeddings into a format suitable for Milvus embeddings = sentence_embeddings.numpy().tolist()In this section, we're using the transformer model to tokenize the sentences and generate their embeddings. We then normalize these embeddings and convert them to a format suitable for insertion into Milvus (Python lists).With the pooling function defined, we're now equipped to generate the actual sentence embeddings. These embeddings will then be processed and made ready for insertion into Milvus.Inserting vector embeddings into MilvusWe're now ready to interact with Milvus. In this section, we will connect to our locally deployed Milvus server, define a schema for our data, and create a collection in the Milvus database to store our sentence embeddings.Now, it's time to put our Milvus deployment to use. We will define the structure of our data, set up a connection to the Milvus server, and prepare our data for insertion:# Establish a connection to the Milvus server connections.connect("default", host="localhost", port="19530") # Define the schema for our collection fields = [    FieldSchema(name="pk", dtype=DataType.INT64, is_primary=True, auto_id=True),    FieldSchema(name="sentences", dtype=DataType.VARCHAR, is_primary=False, description="The actual sentences",                max_length=256),    FieldSchema(name="embeddings", dtype=DataType.FLOAT_VECTOR, is_primary=False, description="The sentence embeddings",                dim=sentence_embeddings.size()[1]) ] schema = CollectionSchema(fields, "A collection to store sentence embeddings")We establish a connection to the Milvus server and then define the schema for our collection in Milvus. The schema includes a primary key field, a field for the sentences, and a field for the sentence embeddings.With our connection established and schema defined, we can now create our collection, insert our data, and build an index to enable efficient search operations.Create Collection, Insert Data, and Create IndexIn this snippet, we first create the collection in Milvus using the previously defined schema. We then organize our data to match our collection's schema and insert it into our collection. After the data is inserted, we create an index on the embeddings to optimize search operations. Finally, we print the number of entities in the collection to confirm the insertion was successful:# Create the collection in Milvus sentence_embeddings_collection = Collection("sentence_embeddings", schema) # Organize our data to match our collection's schema entities = [    sentences,  # The actual sentences    embeddings,  # The sentence embeddings ] # Insert our data into the collection insert_result = sentence_embeddings_collection.insert(entities) # Create an index to make future search queries faster index = {    "index_type": "IVF_FLAT",    "metric_type": "L2",    "params": {"nlist": 128}, } sentence_embeddings_collection.create_index("embeddings", index) print(f"Number of entities in Milvus: {sentence_embeddings_collection.num_entities}")We create a collection in Milvus using the previously defined schema. We organize our data ( sentences, and sentence embeddings) and insert this data into the collection. Primary keys are generated as auto IDs so we don't need to add them. Finally, we print the number of entities in the collection:This way, the sentences, and their corresponding embeddings are stored in a Milvus collection, ready to be used for similarity searches or other tasks.Now that we've stored our embeddings in Milvus, let's make use of them. We will search for similar vectors in our collection based on similarity to sample vectors.Search Based on Vector SimilarityIn the code, we're loading the data from our collection into memory and then defining the vectors that we want to find similar vectors for:# Load the data into memory sentence_embeddings_collection.load()This step is necessary to load the data in our collection into memory before conducting a search or a query. The search parameters are set, specifying the metric to use for calculating similarity (L2 distance in this case) and the number of clusters to examine during the search operation. The search operation is then performed, and the results are printed out:# Vectors to search vectors_to_search = embeddings[-2:] search_params = {    "metric_type": "L2",    "params": {"nprobe": 10}, } # Perform the search result = sentence_embeddings_collection.search(vectors_to_search, "embeddings", search_params, limit=3, output_fields=["sentences"]) # Print the search results for hits in result:    for hit in hits:        print(f"hit: {hit}, sentence field: {hit.entity.get('sentences')}")Here, we're searching for the two most similar sentence embeddings to the last two embeddings in our list. The results are limited to the top 3 matches, and the corresponding sentences of these matches are printed out:Once we're done with our data, it's a good practice to clean up. In this section, we'll explore how to delete entities from our collection using their primary keys.Delete Entities by Primary KeyThis code first gets the primary keys of the entities that we want to delete. We then query the collection before the deletion operation to show the entities that will be deleted. The deletion operation is performed, and the same query is run after the deletion operation to confirm that the entities have been deleted:# Get the primary keys of the entities we want to delete ids = insert_result.primary_keys expr = f'pk in [{ids[0]}, {ids[1]}]' # Query before deletion result = sentence_embeddings_collection.query(expr=expr, output_fields=["sentences", "embeddings"]) print(f"Query before delete by expr=`{expr}` -> result: \\\\n-{result[0]}\\\\n-{result[1]}\\\\n") # Delete entities sentence_embeddings_collection.delete(expr) # Query after deletion result = sentence_embeddings_collection.query(expr=expr, output_fields=["sentences", "embeddings"]) print(f"Query after delete by expr=`{expr}` -> result: {result}\\\\n")Here, we're deleting the entities corresponding to the first two primary keys in our collection. Before and after the deletion, we perform a query to see the result of the deletion operation:Finally, we drop the entire collection from the Milvus server:# Drop the collection utility.drop_collection("sentence_embeddings")This code first gets the primary keys of the entities that we want to delete. We then query the collection before the deletion operation to show the entities that will be deleted. The deletion operation is performed, and the same query is run after the deletion operation to confirm that the entities have been deleted.ConclusionCongratulations on completing this hands-on tutorial with Milvus! You've learned how to harness the power of an open-source vector database that simplifies and accelerates AI and ML applications. Throughout this journey, you set up Milvus locally using Docker Compose, transformed sentences into high-dimensional embeddings and conducted vector similarity searches for practical use cases.Milvus' advanced indexing techniques have empowered you to efficiently store, search, and analyze large volumes of vector data. Its user-friendly design and seamless integration capabilities ensure that you can leverage its powerful features without unnecessary complexity.As you continue exploring Milvus, you'll uncover even more possibilities for its application in diverse fields, such as recommendation systems, image recognition, and natural language processing. The high-performance similarity search and analytics offered by Milvus open doors to cutting-edge AI-driven solutions.With your newfound expertise in Milvus, you are equipped to embark on your own AI adventures, leveraging the potential of vector databases to tackle real-world challenges. Continue experimenting, innovating, and building AI-driven applications that push the boundaries of what's possible. Happy coding!Author Bio:Alan Bernardo Palacio is a data scientist and an engineer with vast experience in different engineering fields. His focus has been the development and application of state-of-the-art data products and algorithms in several industries. He has worked for companies such as Ernst and Young, Globant, and now holds a data engineer position at Ebiquity Media helping the company to create a scalable data pipeline. Alan graduated with a Mechanical Engineering degree from the National University of Tucuman in 2015, participated as the founder in startups, and later on earned a Master's degree from the faculty of Mathematics in the Autonomous University of Barcelona in 2017. Originally from Argentina, he now works and resides in the Netherlands.LinkedIn 

Dive deeper into the world of AI innovation and stay ahead of the AI curve! Subscribe to our AI_Distilled newsletter for the latest insights and books. Don't miss out – sign up today!IntroductionData analysis plays a crucial role in extracting meaningful insights from raw data, driving informed decision-making in various fields. Python's Pandas library has long been a go-to tool for data manipulation and analysis. Now, imagine enhancing Pandas with the power of Generative AI, enabling data analysis to become conversational and intuitive. Enter PandasAI, a Python library that seamlessly integrates Generative AI capabilities into Pandas, revolutionizing the way we interact with data.PandasAI is designed to bridge the gap between traditional data analysis workflows and the realm of artificial intelligence. By combining the strengths of Pandas and Generative AI, PandasAI empowers users to engage in natural language conversations with their data. This innovative library brings a new level of interactivity and flexibility to the data analysis process.With PandasAI, you can effortlessly pose questions to your dataset using human-like language, transforming complex queries into simple conversational statements. The library leverages machine learning models to interpret and understand these queries, intelligently extracting the desired insights from the data. This conversational approach eliminates the need for complex syntax and allows users, regardless of their technical background, to interact with data in a more intuitive and user-friendly way.Under the hood, PandasAI combines the power of natural language processing (NLP) and machine learning techniques. By leveraging pre-trained models, it infers user intent, identifies relevant data patterns, and generates insightful responses. Furthermore, PandasAI supports a wide range of data analysis operations, including data cleaning, aggregation, visualization, and more. It seamlessly integrates with existing Pandas workflows, making it a versatile and valuable addition to any data scientist or analyst's toolkit.In this comprehensive blog post, we will first cover how to install and configure PandasAI, followed by detailed usage examples to demonstrate its capabilities.Installing and Configuring PandasAIPandasAI can be easily installed using pip, Python's package manager:pip install pandasaiThis will download and install the latest version of the PandasAI package along with any required dependencies.Next, you need to configure credentials for the AI engine that will power PandasAI's NLP capabilities:from pandasai.llm.openai import OpenAI openai_api_key = "sk-..." llm = OpenAI(api_token=openai_api_key) ai = PandasAI(llm)PandasAI offers detailed documentation on how to get API keys for services like OpenAI and Anthropic.Once configured, PandasAI is ready to supercharge your data tasks through the power of language. Let's now see it in action through some examples.Intuitive Data Exploration Using Natural LanguageA key strength of PandasAI is enabling intuitive data exploration using plain English. Consider this sample data:data = pd.DataFrame({    'Product': ['A', 'B', 'C'],    'Sales': [100, 200, 50],    'Region': ['East', 'West', 'West']}) ai.init(data)You can now ask questions about this data conversationally:ai.run("Which region had the highest sales?") ai.run("Plot sales by product as a bar chart ordered by sales")PandasAI will automatically generate relevant summaries, plots, and insights from the data based on the natural language prompts.Automating Complex Multi-Step Data PipelinesPandasAI also excels at automating relatively complex multi-step analytical data workflows: ai.run("""    Load sales and inventory data    Join tables on product_id    Impute missing values    Remove outliers    Calculate inventory turnover ratio    Segment products into ABC categories """)This eliminates tedious manual coding effort with Pandas.Unified Analysis across Multiple DatasetsFor real-world analysis, PandasAI can work seamlessly across multiple datasets:sales = pd.read_csv("sales.csv") product = pd.read_csv("product.csv") customer = pd.read_csv("customer.csv") ai.add_frames(sales, product, customer) ai.run("Join the datasets. Show average order size by customer city.")This enables deriving unified insights across disconnected data sources.Building Data-Driven Analytics ApplicationsBeyond exploration, PandasAI can power analytics apps via Python integration. For instance:region = input("Enter region: ") ai.run(f"Compare {region} sales to national average") This allows creating customizable analytics tools for business users tailored to their needs. PandasAI can also enable production apps using Streamlit for the UI: import streamlit as st from pandasai import PandasAI region = st.text_input("Enter region:") … … … if region:    insight = ai.run(f"Analyze {region} sales")    st.write(insight)Democratizing Data-Driven DecisionsA key promise of PandasAI is democratizing data analysis by removing coding complexity. This allows non-technical users to independently extract insights through natural language.Data-driven decisions can become decentralized rather than relying on centralized analytics teams. Domain experts can get tailored insights on demand without coding expertise.Real-World ApplicationsLet's explore some real-world applications of PandasAI to understand how it can benefit various industries:FinanceFinancial analysts can use PandasAI to quickly analyze stock market data, generate investment insights, and create financial reports. They can ask questions like, "What are the top-performing stocks in the last quarter?" and receive instant answers. For Example:import pandas as pd from pandasai import PandasAI stocks = pd.read_csv("stocks.csv") ai = PandasAI(model="codex") ai.init(stocks) ai.run("What were the top 5 performing stocks last quarter?") ai.run("Compare revenue growth across technology and healthcare stocks") ai.run("Which sectors saw the most upside surprises in earnings last quarter?")HealthcareHealthcare professionals can leverage PandasAI to analyze patient data, track disease trends, and make informed decisions about patient care. They can ask questions like, "What are the common risk factors for a particular disease?" and gain valuable insights.MarketingMarketers can use PandasAI to analyze customer data, segment audiences, and optimize marketing strategies. They can ask questions like, "Which marketing channels have the highest conversion rates?" and fine-tune their campaigns accordingly.E-commerceE-commerce businesses can benefit from PandasAI by analyzing sales data, predicting customer behavior, and optimizing inventory management. They can ask questions like, "What products are likely to be popular next month?" and plan their stock accordingly.ConclusionPandasAI represents an exciting glimpse into the future of data analysis driven by AI advancement. By automating the tedious parts of data preparation and manipulation, PandasAI allows data professionals to focus on high-value tasks - framing the right questions, interpreting insights, and telling impactful data stories.Its natural language interface also promises to open up data exploration and analysis to non-technical domain experts. Rather than writing code, anyone can derive tailored insights from data by simply asking questions in plain English.As AI continues progressing, we can expect PandasAI to become even more powerful and nuanced in its analytical abilities over time. It paves the path for taking data science from simple pattern recognition to deeper knowledge generation using machines that learn, reason and connect concepts.While early in its development, PandasAI offers a taste of what is possible when the foundations of data analysis are reimagined using AI. It will be fascinating to see how this library helps shape and transform the analytics landscape in the coming years. For forward-thinking data professionals, the time to embrace its possibilities is now.In summary, by synergizing the strengths of Pandas and large language models, PandasAI promises to push the boundaries of what is possible in data analysis today. It represents an important milestone in the AI-driven evolution of the field.Author BioRohan Chikorde is an accomplished AI Architect professional with a post-graduate in Machine Learning and Artificial Intelligence. With almost a decade of experience, he has successfully developed deep learning and machine learning models for various business applications. Rohan's expertise spans multiple domains, and he excels in programming languages such as R and Python, as well as analytics techniques like regression analysis and data mining. In addition to his technical prowess, he is an effective communicator, mentor, and team leader. Rohan's passion lies in machine learning, deep learning, and computer vision.LinkedIn

In this article by Gion Kunz, author of the book Mastering Angular 2 Components, has explained the concept of directives from the first version of Angular changed the game in frontend UI frameworks. This was the first time that I felt that there was a simple yet powerful concept that allowed the creation of reusable UI components. Directives could communicate with DOM events or messaging services. They allowed you to follow the principle of composition, and you could nest directives and create larger directives that solely consisted of smaller directives arranged together. Actually, directives were a very nice implementation of components for the browser. (For more resources related to this topic, see here.) In this section, we'll look into the component-based architecture of Angular 2 and how the previous topic about general UI components will fit into Angular. Everything is a component As an early adopter of Angular 2 and while talking to other people about it, I got frequently asked what the biggest difference is to the first version. My answer to this question was always the same. Everything is a component. For me, this paradigm shift was the most relevant change that both simplified and enriched the framework. Of course, there are a lot of other changes with Angular 2. However, as an advocate of component-based user interfaces, I've found that this change is the most interesting one. Of course, this change also came with a lot of architectural changes. Angular 2 supports the idea of looking at the user interface holistically and supporting composition with components. However, the biggest difference to its first version is that now your pages are no longer global views, but they are simply components that are assembled from other components. If you've been following this chapter, you'll notice that this is exactly what a holistic approach to user interfaces demands. No more pages but systems of components. Angular 2 still uses the concept of directives, although directives are now really what the name suggests. They are orders for the browser to attach a given behavior to an element. Components are a special kind of directives that come with a view. Creating a tabbed interface component Let's introduce a new UI component in our ui folder in the project that will provide us with a tabbed interface that we can use for composition. We use what we learned about content projection in order to make this component reusable. We'll actually create two components, one for Tabs, which itself holds individual Tab components. First, let's create the component class within a new tabs/tab folder in a file called tab.js: import {Component, Input, ViewEncapsulation, HostBinding} from '@angular/core'; import template from './tab.html!text'; @Component({ selector: 'ngc-tab', host: { class: 'tabs__tab' }, template, encapsulation: ViewEncapsulation.None }) export class Tab { @Input() name; @HostBinding('class.tabs__tab--active') active = false; } The only state that we store in our Tab component is whether the tab is active or not. The name that is displayed on the tab will be available through an input property. We use a class property binding to make a tab visible. Based on the active flag we set a class; without this, our tabs are hidden. Let's take a look at the tab.html template file of this component: <ng-content></ng-content> This is it already? Actually, yes it is! The Tab component is only responsible for the storage of its name and active state, as well as the insertion of the host element content in the content projection point. There's no additional templating that is needed. Now, we'll move one level up and create the Tabs component that will be responsible for the grouping all the Tab components. As we won't include Tab components directly when we want to create a tabbed interface but use the Tabs component instead, this needs to forward content that we put into the Tabs host element. Let's look at how we can achieve this. In the tabs folder, we will create a tabs.js file that contains our Tabs component code, as follows: import {Component, ViewEncapsulation, ContentChildren} from '@angular/core'; import template from './tabs.html!text'; // We rely on the Tab component import {Tab} from './tab/tab'; @Component({ selector: 'ngc-tabs', host: { class: 'tabs' }, template, encapsulation: ViewEncapsulation.None, directives: [Tab] }) export class Tabs { // This queries the content inside <ng-content> and stores a // query list that will be updated if the content changes @ContentChildren(Tab) tabs; // The ngAfterContentInit lifecycle hook will be called once the // content inside <ng-content> was initialized ngAfterContentInit() { this.activateTab(this.tabs.first); } activateTab(tab) { // To activate a tab we first convert the live list to an // array and deactivate all tabs before we set the new // tab active this.tabs.toArray().forEach((t) => t.active = false); tab.active = true; } } Let's observe what's happening here. We used a new @ContentChildren annotation, in order to query our inserted content for directives that match the type that we pass to the decorator. The tabs property will contain an object of the QueryList type, which is an observable list type that will be updated if the content projection changes. You need to remember that content projection is a dynamic process as the content in the host element can actually change, for example, using the NgFor or NgIf directives. We use the AfterContentInit lifecycle hook, which we've already briefly discussed in the Custom UI elements section of Chapter 2, Ready, Set, Go! This lifecycle hook is called after Angular has completed content projection on the component. Only then we have the guarantee that our QueryList object will be initialized, and we can start working with child directives that were projected as content. The activateTab function will set the Tab components active flag, deactivating any previous active tab. As the observable QueryList object is not a native array, we first need to convert it using toArray() before we start working with it. Let's now look at the template of the Tabs component that we created in a file called tabs.html in the tabs directory: <ul class="tabs__tab-list"> <li *ngFor="let tab of tabs"> <button class="tabs__tab-button" [class.tabs__tab-button--active]="tab.active" (click)="activateTab(tab)">{{tab.name}}</button> </li> </ul> <div class="tabs__l-container"> <ng-content select="ngc-tab"></ng-content> </div> The structure of our Tabs component is as follows. First we render all the tab buttons in an unordered list. After the unordered list, we have a tabs container that will contain all our Tab components that are inserted using content projection and the <ng-content> element. Note that the selector that we use is actually the selector we use for our Tab component. Tabs that are not active will not be visible because we control this using CSS on our Tab component class attribute binding (refer to the Tab component code). This is all that we need to create a flexible and well-encapsulated tabbed interface component. Now, we can go ahead and use this component in our Project component to provide a segregation of our project detail information. We will create three tabs for now where the first one will embed our task list. We will address the content of the other two tabs in a later chapter. Let's modify our Project component template in the project.html file as a first step. Instead of including our TaskList component directly, we now use the Tabs and Tab components to nest the task list into our tabbed interface: <ngc-tabs> <ngc-tab name="Tasks"> <ngc-task-list [tasks]="tasks" (tasksUpdated)="updateTasks($event)"> </ngc-task-list> </ngc-tab> <ngc-tab name="Comments"></ngc-tab> <ngc-tab name="Activities"></ngc-tab> </ngc-tabs> You should have noticed by now that we are actually nesting two components within this template code using content projection, as follows: First, the Tabs component uses content projection to select all the <ngc-tab> elements. As these elements happen to be components too (our Tab component will attach to elements with this name), they will be recognized as such within the Tabs component once they are inserted. In the <ngc-tab> element, we then nest our TaskList component. If we go back to our Task component template, which will be attached to elements with the name ngc-tab, we will have a generic projection point that inserts any content that is present in the host element. Our task list will effectively be passed through the Tabs component into the Tab component. The visual efforts timeline Although the components that we created so far to manage efforts provide a good way to edit and display effort and time durations, we can still improve this with some visual indication. In this section, we will create a visual efforts timeline using SVG. This timeline should display the following information: The total estimated duration as a grey background bar The total effective duration as a green bar that overlays on the total estimated duration bar A yellow bar that shows any overtime (if the effective duration is greater than the estimated duration) The following two figures illustrate the different visual states of our efforts timeline component: The visual state if the estimated duration is greater than the effective duration The visual state if the effective duration exceeds the estimated duration (the overtime is displayed as a yellow bar) Let's start fleshing out our component by creating a new EffortsTimeline Component class on the lib/efforts/efforts-timeline/efforts-timeline.js path: … @Component({ selector: 'ngc-efforts-timeline', … }) export class EffortsTimeline { @Input() estimated; @Input() effective; @Input() height; ngOnChanges(changes) { this.done = 0; this.overtime = 0; if (!this.estimated && this.effective || (this.estimated && this.estimated === this.effective)) { // If there's only effective time or if the estimated time // is equal to the effective time we are 100% done this.done = 100; } else if (this.estimated < this.effective) { // If we have more effective time than estimated we need to // calculate overtime and done in percentage this.done = this.estimated / this.effective * 100; this.overtime = 100 - this.done; } else { // The regular case where we have less effective time than // estimated this.done = this.effective / this.estimated * 100; } } } Our component has three input properties: estimated: This is the estimated time duration in milliseconds effective: This is the effective time duration in milliseconds height: This is the desired height of the efforts timeline in pixels In the OnChanges lifecycle hook, we set two component member fields, which are based on the estimated and effective time: done: This contains the width of the green bar in percentage that displays the effective duration without overtime that exceeds the estimated duration overtime: This contains the width of the yellow bar in percentage that displays any overtime, which is any time duration that exceeds the estimated duration Let's look at the template of the EffortsTimeline component and see how we can now use the done and overtime member fields to draw our timeline. We will create a new lib/efforts/efforts-timeline/efforts-timeline.html file: <svg width="100%" [attr.height]="height"> <rect [attr.height]="height" x="0" y="0" width="100%" class="efforts-timeline__remaining"></rect> <rect *ngIf="done" x="0" y="0" [attr.width]="done + '%'" [attr.height]="height" class="efforts-timeline__done"></rect> <rect *ngIf="overtime" [attr.x]="done + '%'" y="0" [attr.width]="overtime + '%'" [attr.height]="height" class="efforts-timeline__overtime"></rect> </svg> Our template is SVG-based, and it contains three rectangles for each of the bars that we want to display. The background bar that will be visible if there is remaining effort will always be displayed. Above the remaining bar, we conditionally display the done and the overtime bar using the calculated widths from our component class. Now, we can go ahead and include the EffortsTimeline class in our Efforts component. This way our users will have visual feedback when they edit the estimated or effective duration, and it provides them a sense of overview. Let's look into the template of the Efforts component to see how we integrate the timeline: … <ngc-efforts-timeline height="10" [estimated]="estimated" [effective]="effective"> </ngc-efforts-timeline> As we have the estimated and effective duration times readily available in our Efforts component, we can simply create a binding to the EffortsTimeline component input properties: The Efforts component displaying our newly-created efforts timeline component (the overtime of six hours is visualized with the yellow bar) Summary In this article, we learned about the architecture of the components in Angular. We also learned how to create a tabbed interface component and how to create a visual efforts timeline using SVG. Resources for Article: Further resources on this subject: Angular 2.0[article] AngularJS Project[article] AngularJS[article]