Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7018 Articles
article-image-introduction-network-security
Packt
06 Apr 2017
18 min read
Save for later

Introduction to Network Security

Packt
06 Apr 2017
18 min read
In this article by Warun Levesque, Michael McLafferty, and Arthur Salmon, the authors of the book Applied Network Security, we will be covering the following topics, which will give an introduction to network security: Murphy's law The definition of a hacker and the types The hacking process Recent events and statistics of network attacks Security for individual versus company Mitigation against threats Building an assessment This world is changing rapidly with advancing network technologies. Unfortunately, sometimes the convenience of technology can outpace its security and safety. Technologies like the Internet of things is ushering in a new era of network communication. We also want to change the mindset of being in the field of network security. Most current cyber security professionals practice defensive and passive security. They mostly focus on mitigation and forensic tactics to analyze the aftermath of an attack. We want to change this mindset to one of Offensive security. This article will give insight on how a hacker thinks and what methods they use. Having knowledge of a hacker's tactics, will give the reader a great advantage in protecting any network from attack. (For more resources related to this topic, see here.) Murphy's law Network security is much like Murphy's law in the sense that, if something can go wrong it will go wrong. To be successful at understanding and applying network security, a person must master the three Ps. The three Ps are, persistence, patience, and passion. A cyber security professional must be persistent in their pursuit of a solution to a problem. Giving up is not an option. The answer will be there; it just may take more time than expected to find it. Having patience is also an important trait to master. When dealing with network anomalies, it is very easy to get frustrated. Taking a deep breath and keeping a cool head goes a long way in finding the correct solution to your network security problems. Finally, developing a passion for cyber security is critical to being a successful network security professional. Having that passion will drive you to learn more and evolve yourself on a daily basis to be better. Once you learn, then you will improve and perhaps go on to inspire others to reach similar aspirations in cyber security. The definition of a hacker and the types A hacker is a person who uses computers to gain unauthorized access to data. There are many different types of hackers. There are white hat, grey hat, and black hat hackers. Some hackers are defined for their intention. For example, a hacker that attacks for political reasons may be known as a hacktivist. A white hat hackers have no criminal intent, but instead focuses on finding and fixing network vulnerabilities. Often companies will hire a white hat hacker to test the security of their network for vulnerabilities. A grey hat hacker is someone who may have criminal intent but not often for personal gain. Often a grey hat will seek to expose a network vulnerability without the permission from the owner of the network. A black hat hacker is purely criminal. They sole objective is personal gain. Black hat hackers take advantage of network vulnerabilities anyway they can for maximum benefit. A cyber-criminal is another type of black hat hacker, who is motivated to attack for illegal financial gain. A more basic type of hacker is known as a script kiddie. A script kiddie is a person who knows how to use basic hacking tools but doesn't understand how they work. They often lack the knowledge to launch any kind of real attack, but can still cause problems on a poorly protected network. Hackers tools There are a range of many different hacking tools. A tool like Nmap for example, is a great tool for both reconnaissance and scanning for network vulnerabilities. Some tools are grouped together to make toolkits and frameworks, such as the Social Engineering Toolkit and Metasploit framework. The Metasploit framework is one of the most versatile and supported hacking tool frameworks available. Metasploit is built around a collection of highly effective modules, such as MSFvenom and provides access to an extensive database of exploits and vulnerabilities. There are also physical hacking tools. Devices like the Rubber Ducky and Wi-Fi Pineapple are good examples. The Rubber Ducky is a usb payload injector, that automatically injects a malicious virus into the device it's plugged into. The Wi-Fi Pineapple can act as a rogue router and be used to launch man in the middle attacks. The Wi-Fi Pineapple also has a range of modules that allow it to execute multiple attack vectors. These types of tools are known as penetration testing equipment. The hacking process There are five main phases to the hacking process: Reconnaissance: The reconnaissance phase is often the most time consuming. This phase can last days, weeks, or even months sometimes depending on the target. The objective during the reconnaissance phase is to learn as much as possible about the potential target. Scanning: In this phase the hacker will scan for vulnerabilities in the network to exploit. These scans will look for weaknesses such as, open ports, open services, outdated applications (including operating systems), and the type of equipment being used on the network. Access: In this phase the hacker will use the knowledge gained in the previous phases to gain access to sensitive data or use the network to attack other targets. The objective of this phase is to have the attacker gain some level of control over other devices on the network. Maintaining access: During this phase a hacker will look at various options, such as creating a backdoor to maintain access to devices they have compromised. By creating a backdoor, a hacker can maintain a persistent attack on a network, without fear of losing access to the devices they have gained control over. Although when a backdoor is created, it increases the chance of a hacker being discovered. Backdoors are noisy and often leave a large footprint for IDS to follow. Covering your tracks: This phase is about hiding the intrusion of the network by the hacker as to not alert any IDS that may be monitoring the network. The objective of this phase is to erase any trace that an attack occurred on the network. Recent events and statistics of network attacks The news has been full of cyber-attacks in recent years. The number and scale of attacks are increasing at an alarming rate. It is important for anyone in network security to study these attacks. Staying current with this kind of information will help in defending your network from similar attacks. Since 2015, the medical and insurance industry have been heavily targeted for cyber-attacks. On May 5th, 2015 Premera Blue Cross was attacked. This attack is said to have compromised at least 11 million customer accounts containing personal data. The attack exposed customer's names, birth dates, social security numbers, phone numbers, bank account information, mailing, and e-mail addresses. Another attack that was on a larger scale, was the attack on Anthem. It is estimated that 80 million personal data records were stolen from customers, employees, and even the Chief Executive Officer of Anthem. Another more infamous cyber-attack recently was the Sony hack. This hack was a little different than the Anthem and Blue Cross attacks, because it was carried out by hacktivist instead of cyber-criminals. Even though both types of hacking are criminal, the fundamental reasoning and objectives of the attacks are quite different. The objective in the Sony attack was to disrupt and embarrass the executives at Sony as well as prevent a film from being released. No financial data was targeted. Instead the hackers went after personal e-mails of top executives. The hackers then released the e-mails to the public, causing humiliation to Sony and its executives. Many apologies were issued by Sony in the following weeks of the attack. Large commercial retailers have also been a favorite target for hackers. An attack occurred against Home Depot in September 2014. That attack was on a large scale. It is estimated that over 56 million credit cards were compromised during the Home Depot attack. A similar attack but on a smaller scale was carried out against Staples in October 2014. During this attack, over 1.4 million credit card numbers were stolen. The statistics on cyber security attacks are eye opening. It is estimated by some experts that cybercrime has a worldwide cost of 110 billion dollars a year. In a given year, over 15 million Americans will have their identified stolen through cyber-attacks, it is also estimated that 1.5 million people fall victim to cybercrime every day. These statistics are rapidly increasing and will continue to do so until more people take an active interest in network security. Our defense The baseline for preventing a potential security issues typically begins with hardening the security infrastructure, including firewalls, DMZ, and physical security platform. Entrusting only valid sources or individuals with personal data and or access to that data. That also includes being compliant with all regulations that apply to a given situation or business. Being aware of the types of breaches as well as your potential vulnerabilities. Also understanding is an individual or an organization a higher risk target for attacks. The question has to be asked, does one's organization promote security? This is done both at the personal and the business level to deter cyber-attacks? After a decade of responding to incidents and helping customers recover from and increase their resilience against breaches. Organization may already have a security training and awareness (STA) program, or other training and program could have existed. As the security and threat landscape evolves organizations and individuals need to continually evaluate practices that are required and appropriate for the data they collect, transmit, retain, and destroy. Encryption of data at rest/in storage and in transit is a fundamental security requirement and the respective failure is frequently being cited as the cause for regulatory action and lawsuits. Enforce effective password management policies. Least privilege user access (LUA) is a core security strategy component, and all accounts should run with as few privileges and access levels as possible. Conduct regular security design and code reviews including penetration tests and vulnerability scans to identify and mitigate vulnerabilities. Require e-mail authentication on all inbound and outbound mail servers to help detect malicious e-mail including spear phishing and spoofed e-mail. Continuously monitor in real-time the security of your organization's infrastructure including collecting and analyzing all network traffic, and analyzing centralized logs (including firewall, IDS/IPS, VPN, and AV) using log management tools, as well as reviewing network statistics. Identify anomalous activity, investigate, and revise your view of anomalous activity accordingly. User training would be the biggest challenge, but is arguably the most important defense. Security for individual versus company One of the fundamental questions individuals need to ask themselves, is there a difference between them the individual and an organization? Individual security is less likely due to attack service area. However, there are tools and sites on the internet that can be utilized to detect and mitigate data breaches for both.https://haveibeenpwned.com/ or http://map.norsecorp.com/ are good sites to start with. The issue is that individuals believe they are not a target because there is little to gain from attacking individuals, but in truth everyone has the ability to become a target. Wi-Fi vulnerabilities Protecting wireless networks can be very challenging at times. There are many vulnerabilities that a hacker can exploit to compromise a wireless network. One of the basic Wi-Fi vulnerabilities is broadcasting the Service Set Identifier (SSID) of your wireless network. Broadcasting the SSID makes the wireless network easier to find and target. Another vulnerability in Wi-Fi networks is using Media Access Control (MAC) addressesfor network authentication. A hacker can easily spoof or mimic a trusted MAC address to gain access to the network. Using weak encryption such as Wired Equivalent Privacy (WEP) will make your network an easy target for attack. There are many hacking tools available to crack any WEP key in under five minutes. A major physical vulnerability in wireless networks are the Access Points (AP). Sometimes APs will be placed in poor locations that can be easily accessed by a hacker. A hacker may install what is called a rogue AP. This rogue AP will monitor the network for data that a hacker can use to escalate their attack. Often this tactic is used to harvest the credentials of high ranking management personnel, to gain access to encrypted databases that contain the personal/financial data of employees and customers or both. Peer-to-peer technology can also be a vulnerability for wireless networks. A hacker may gain access to a wireless network by using a legitimate user as an accepted entry point. Not using and enforcing security policies is also a major vulnerability found in wireless networks. Using security tools like Active Directory (deployed properly) will make it harder for a hacker to gain access to a network. Hackers will often go after low hanging fruit (easy targets), so having at least some deterrence will go a long way in protecting your wireless network. Using Intrusion Detection Systems (IDS) in combination with Active Directory will immensely increase the defense of any wireless network. Although the most effective factor is, having a well-trained and informed cyber security professional watching over the network. The more a cyber security professional (threat hunter) understands the tactics of a hacker, the more effective that Threat hunter will become in discovering and neutralizing a network attack. Although there are many challenges in protecting a wireless network, with the proper planning and deployment those challenges can be overcome. Knowns and unknowns The toughest thing about an unknown risk to security is that they are unknown. Unless they are found they can stay hidden. A common practice to determine an unknown risk would be to identify all the known risks and attempt to mitigate them as best as possible. There are many sites available that can assist in this venture. The most helpful would be reports from CVE sites that identify vulnerabilities. False positives   Positive Negative True TP: correctly identified TN: correctly rejected False FP: incorrectly identified FN: incorrectly rejected As it related to detection for an analyzed event there are four situations that exist in this context, corresponding to the relation between the result of the detection for an analyzed event. In this case, each of the corresponding situations mentioned in the preceding table are outlined as follows: True positive (TP): It is when the analyzed event is correctly classified as intrusion or as harmful/malicious. For example, a network security administrator enters their credentials into the Active Directory server and is granted administrator access. True negative (TN): It is when the analyzed event is correctly classified and correctly rejected. For example, an attacker uses a port like 4444 to communicate with a victim's device. An intrusion detection system detects network traffic on the authorized port and alerts the cyber security team to this potential malicious activity. The cyber security team quickly closes the port and isolates the infected device from the network. False positive (FP): It is when the analyzed event is innocuous or otherwise clean as it relates to perspective of security, however, the system classifies it as malicious or harmful. For example, a user types their password into a website's login text field. Instead of being granted access, the user is flagged for an SQL injection attempt by input sanitation. This is often caused when input sanitation is misconfigured. False negative (FN): It is when the analyzed event is malicious but it is classified as normal/innocuous. For example, an attacker inputs an SQL injection string into a text field found on a website to gain unauthorized access to database information. The website accepts the SQL injection as normal user behavior and grants access to the attacker. As it relates to detection, having systems correctly identify the given situations in paramount. Mitigation against threats There are many threats that a network faces. New network threats are emerging all the time. As a network security, professional, it would be wise to have a good understanding of effective mitigation techniques. For example, a hacker using a packet sniffer can be mitigated by only allowing the network admin to run a network analyzer (packet sniffer) on the network. A packet sniffer can usually detect another packet sniffer on the network right away. Although, there are ways a knowledgeable hacker can disguise the packet sniffer as another piece of software. A hacker will not usually go to such lengths unless it is a highly-secured target. It is alarming that; most businesses do not properly monitor their network or even at all. It is important for any business to have a business continuity/disaster recovery plan. This plan is intended to allow a business to continue to operate and recover from a serious network attack. The most common deployment of the continuity/disaster recovery plan is after a DDoS attack. A DDoS attack could potentially cost a business or organization millions of dollars is lost revenue and productivity. One of the most effective and hardest to mitigate attacks is social engineering. All the most devastating network attacks have begun with some type of social engineering attack. One good example is the hack against Snapchat on February 26th, 2016. "Last Friday, Snapchat's payroll department was targeted by an isolated e-mail phishing scam in which a scammer impersonated our Chief Executive Officer and asked for employee payroll information," Snapchat explained in a blog post. Unfortunately, the phishing e-mail wasn't recognized for what it was — a scam — and payroll information about some current and former employees was disclosed externally. Socially engineered phishing e-mails, same as the one that affected Snapchat are common attack vectors for hackers. The one difference between phishing e-mails from a few years ago, and the ones in 2016 is, the level of social engineering hackers are putting into the e-mails. The Snapchat HR phishing e-mail, indicated a high level of reconnaissance on the Chief Executive Officer of Snapchat. This reconnaissance most likely took months. This level of detail and targeting of an individual (Chief Executive Officer) is more accurately know as a spear-phishing e-mail. Spear-phishing campaigns go after one individual (fish) compared to phishing campaigns that are more general and may be sent to millions of users (fish). It is like casting a big open net into the water and seeing what comes back. The only real way to mitigate against social engineering attacks is training and building awareness among users. By properly training the users that access the network, it will create a higher level of awareness against socially engineered attacks. Building an assessment Creating a network assessment is an important aspect of network security. A network assessment will allow for a better understanding of where vulnerabilities may be found within the network. It is important to know precisely what you are doing during a network assessment. If the assessment is done wrong, you could cause great harm to the network you are trying to protect. Before you start the network assessment, you should determine the objectives of the assessment itself. Are you trying to identify if the network has any open ports that shouldn't be? Is your objective to quantify how much traffic flows through the network at any given time or a specific time? Once you decide on the objectives of the network assessment, you will then be able to choose the type of tools you will use. Network assessment tools are often known as penetration testing tools. A person who employs these tools is known as a penetration tester or pen tester. These tools are designed to find and exploit network vulnerabilities, so that they can be fixed before a real attack occurs. That is why it is important to know what you are doing when using penetration testing tools during an assessment. Sometimes network assessments require a team. It is important to have an accurate idea of the scale of the network before you pick your team. In a large enterprise network, it can be easy to become overwhelmed by tasks to complete without enough support. Once the scale of the network assessment is complete, the next step would be to ensure you have written permission and scope from management. All parties involved in the network assessment must be clear on what can and cannot be done to the network during the assessment. After the assessment is completed, the last step is creating a report to educate concerned parties of the findings. Providing detailed information and solutions to vulnerabilities will help keep the network up to date on defense. The report will also be able to determine if there are any viruses lying dormant, waiting for the opportune time to attack the network. Network assessments should be conducted routinely and frequently to help ensure strong networksecurity. Summary In this article we covered the fundamentals of network security. It began by explaining the importance of having network security and what should be done to secure the network. It also covered the different ways physical security can be applied. The importance of having security policies in place and wireless security was discussed. This article also spoke about wireless security policies and why they are important. Resources for Article: Further resources on this subject: API and Intent-Driven Networking [article] Deploying First Server [article] Point-to-Point Networks [article]
Read more
  • 0
  • 0
  • 23871

article-image-18-people-in-tech-every-programmer-and-software-engineer-needs-to-follow-in-2019
Richard Gall
02 Jan 2019
9 min read
Save for later

18 people in tech every programmer and software engineer needs to follow in 2019

Richard Gall
02 Jan 2019
9 min read
After a tumultuous 2018 in tech, it's vital that you surround yourself with a variety of opinions and experiences in 2019 if you're to understand what the hell is going on. While there are thousands of incredible people working in tech, I've decided to make life a little easier for you by bringing together 18 of the best people from across the industry to follow on Twitter. From engineers at Microsoft and AWS, to researchers and journalists, this list is by no means comprehensive but it does give you a wide range of people that have been influential, interesting, and important in 2018. (A few of) the best people in tech on Twitter April Wensel (@aprilwensel) April Wensel is the founder of Compassionate Coding, an organization that aims to bring emotional intelligence and ethics into the tech industry. In April 2018 Wensel wrote an essay arguing that "it's time to retire RTFM" (read the fucking manual). The essay was well received by many in the tech community tired of a culture of ostensible caustic machismo and played a part in making conversations around community accessibility an important part of 2018. Watch her keynote at NodeJS Interactive: https://www.youtube.com/watch?v=HPFuHS6aPhw Liz Fong-Jones (@lizthegrey) Liz Fong-Jones is an SRE and Dev Advocate at Google Cloud Platform, but over the last couple of years she has become an important figure within tech activism. First helping to create the NeverAgain pledge in response to the election of Donald Trump in 2016, then helping to bring to light Google's fraught internal struggle over diversity, Fong-Jones has effectively laid the foundations for the mainstream appearance of tech activism in 2018. In an interview with Fast Company, Fong-Jones says she has accepted her role as a spokesperson for the movement that has emerged, but she's committed to helping to "equipping other employees to fight for change in their workplaces–whether at Google or not –so that I’m not a single point of failure." Ana Medina (@Ana_M_Medina) Ana Medina is a chaos engineer at Gremlin. Since moving to the chaos engineering platform from Uber (where she was part of the class action lawsuit against the company), Medina has played an important part in explaining what chaos engineering looks like in practice all around the world. But she is also an important voice in discussions around diversity and mental health in the tech industry - if you get a chance to her her talk, make sure you take the chance, and if you don't, you've still got Twitter... Sarah Drasner (@sarah_edo) Sarah Drasner does everything. She's a Developer Advocate at Microsoft, part of the VueJS core development team, organizer behind Concatenate (a free conference for Nigerian developers), as well as an author too. https://twitter.com/sarah_edo/status/1079400115196944384 Although Drasner specializes in front end development and JavaScript, she's a great person to follow on Twitter for her broad insights on how we learn and evolve as software developers. Do yourself a favour and follow her. Mark Imbriaco (@markimbriaco) Mark Imbriaco is the technical director at Epic Games. Given the company's truly, er, epic year thanks to Fortnite, Imbriaco can offer an insight on how one of the most important and influential technology companies on the planet are thinking. Corey Quinn (@QuinnyPig) Corey Quinn is an AWS expert. As the brain behind the Last Week in AWS newsletter and the voice behind the Screaming in the Cloud podcast (possibly the best cloud computing podcast on the planet), he is without a doubt the go-to person if you want to know what really matters in cloud. The range of guests that Quinn gets on the podcast is really impressive, and sums up his online persona: open, engaged, and always interesting. Yasmine Evjen (@YasmineEvjen) Yasmine Evjen is a Design Advocate at Google. That means that she is not only one of the minds behind Material Design, she is also someone that is helping to demonstrate the importance of human centered design around the world. As the presenter of Centered, a web series by the Google Design team about the ways human centered design is used for a range of applications. If you haven't seen it, it's well worth a watch. https://www.youtube.com/watch?v=cPBXjtpGuSA&list=PLJ21zHI2TNh-pgTlTpaW9kbnqAAVJgB0R&index=5&t=0s Suz Hinton (@noopkat) Suz Hinton works on IoT programs at Microsoft. That's interesting in itself, but when she's not developing fully connected smart homes (possibly), Hinton also streams code tutorials on Twitch (also as noopkat). Chris Short (@ChrisShort) If you want to get the lowdown on all things DevOps, you could do a lot worse than Chris Short. He boasts outstanding credentials - he's a CNCF ambassador, has experience with Red Hat and Ansible - but more importantly is the quality of his insights. A great place to begin is with DevOpsish, a newsletter Short produces, which features some really valuable discussions on the biggest issues and talking points in the field. Dan Abramov (@dan_abramov) Dan Abramov is one of the key figures behind ReactJS. Along with @sophiebits,@jordwalke, and @sebmarkbage, Abramov is quite literally helping to define front end development as we know it. If you're a JavaScript developer, or simply have any kind of passing interest in how we'll be building front ends over the next decade, he is an essential voice to have on your timeline. As you'd expect from someone that has helped put together one of the most popular JavaScript libraries in the world, Dan is very good at articulating some of the biggest challenges we face as developers and can provide useful insights on how to approach problems you might face, whether day to day or career changing. Emma Wedekind (@EmmaWedekind) As well as working at GoTo Meeting, Emma Wedekind is the founder of Coding Coach, a platform that connects developers to mentors to help them develop new skills. This experience makes Wedekind an important authority on developer learning. And at a time when deciding what to learn and how to do it can feel like such a challenging and complex process, surrounding yourself with people taking those issues seriously can be immensely valuable. Jason Lengstorf (@jlengstorf) Jason Lengstorf is a Developer Advocate at GatsbyJS (a cool project that makes it easier to build projects with React). His writing - on Twitter and elsewhere - is incredibly good at helping you discover new ways of working and approaching problems. Bridget Kromhout (@bridgetkromhout) Bridget Kromhout is another essential voice in cloud and DevOps. Currently working at Microsoft as Principal Cloud Advocate, Bridget also organizes DevOps Days and presents the Arrested DevOps podcast with Matty Stratton and Trevor Hess. Follow Bridget for her perspective on DevOps, as well as her experience in DevRel. Ryan Burgess (@burgessdryan) Netflix hasn't faced the scrutiny of many of its fellow tech giants this year, which means it's easy to forget the extent to which the company is at the cutting edge of technological innovation. This is why it's well worth following Ryan Burgess - as an engineering manager he's well placed to provide an insight on how the company is evolving from a tech perspective. His talk at Real World React on A/B testing user experiences is well worth watching: https://youtu.be/TmhJN6rdm28 Anil Dash (@anildash) Okay, so chances are you probably already follow Anil Dash - he does have half a million followers already, after all - but if you don't follow him, you most definitely should. Dash is a key figure in new media and digital culture, but he's not just another thought leader, he's someone that actually understands what it takes to actually build this stuff. As CEO of Glitch, a platform for building (and 'remixing') cool apps, he's having an impact on the way developers work and collaborate. 6 years ago, Dash wrote an essay called 'The Web We Lost'. In it, he laments how the web was becoming colonized by a handful of companies who built the key platforms on which we communicate and engage with one another online. Today, after a year of protest and controversy, Dash's argument is as salient as ever - it's one of the reasons it's vital that we listen to him. Jessie Frazelle (@jessfraz) Jessie Frazelle is a bit of a superstar. Which shouldn't really be that surprising - she's someone that seems to have a natural ability to pull things apart and put them back together again and have the most fun imaginable while doing it. Formerly part of the core Docker team, Frazelle now works at GitHub, where her knowledge and expertise is helping to develop the next Microsoft-tinged chapter in GitHub's history. I was lucky enough to see Jessie speak at ChaosConf in September - check out her talk: https://youtu.be/1hhVS4pdrrk Rachel Coldicutt (@rachelcoldicutt) Rachel Coldicutt is the CEO of Doteveryone, a think tank based in the U.K. that champions responsible tech. If you're interested in how technology interacts with other aspects of society and culture, as well as how it is impacting and being impacted by policymakers, Coldicutt is a vital person to follow. Kelsey Hightower (@kelseyhightower) Kelsey Hightower is another superstar in the tech world - when he talks, you need to listen. Hightower currently works at Google Cloud, but he spends a lot of time at conferences evangelizing for more effective cloud native development. https://twitter.com/mattrickard/status/1073285888191258624 If you're interested in anything infrastructure or cloud related, you need to follow Kelsey Hightower. Who did I miss? That's just a list of a few people in tech I think you should follow in 2019 - but who did I miss? Which accounts are essential? What podcasts and newsletters should we subscribe to?
Read more
  • 0
  • 0
  • 23868

article-image-neural-network-architectures-101-understanding-perceptrons
Kunal Chaudhari
12 Feb 2018
9 min read
Save for later

Neural Network Architectures 101: Understanding Perceptrons

Kunal Chaudhari
12 Feb 2018
9 min read
[box type="note" align="" class="" width=""]This article is an excerpt taken from a book Neural Network Programming with Java Second Edition written by Fabio M. Soares and Alan M. F. Souza. This book is for Java developers who want to master developing smarter applications like weather forecasting, pattern recognition etc using neural networks. [/box] In this article we will discuss about perceptrons along with their features, applications and limitations. Perceptrons are a very popular neural network architecture that implements supervised learning. Projected by Frank Rosenblatt in 1957, it has just one layer of neurons, receiving a set of inputs and producing another set of outputs. This was one of the first representations of neural networks to gain attention, especially because of their simplicity. In our Java implementation, this is illustrated with one neural layer (the output layer). The following code creates a perceptron with three inputs and two outputs, having the linear function at the output layer: int numberOfInputs=3; int numberOfOutputs=2; Linear outputAcFnc = new Linear(1.0); NeuralNet perceptron = new NeuralNet(numberOfInputs,numberOfOutputs,             outputAcFnc); Applications and limitations However, scientists did not take long to conclude that a perceptron neural network could only be applied to simple tasks, according to that simplicity. At that time, neural networks were being used for simple classification problems, but perceptrons usually failed when faced with more complex datasets. Let's illustrate this with a very basic example (an AND function) to understand better this issue. Linear separation The example consists of an AND function that takes two inputs, x1 and x2. That function can be plotted in a two-dimensional chart as follows: And now let's examine how the neural network evolves the training using the perceptron rule, considering a pair of two weights, w1 and w2, initially 0.5, and bias valued 0.5 as well. Assume learning rate η equals 0.2: Epoch x1 x2 w1 w2 b y t E Δw1 Δw2 Δb 1 0 0 0.5 0.5 0.5 0.5 0 -0.5 0 0 -0.1 1 0 1 0.5 0.5 0.4 0.9 0 -0.9 0 -0.18 -0.18 1 1 0 0.5 0.32 0.22 0.72 0 -0.72 -0.144 0 -0.144 1 1 1 0.356 0.32 0.076 0.752 1 0.248 0.0496 0.0496 0.0496 2 0 0 0.406 0.370 0.126 0.126 0 -0.126 0.000 0.000 -0.025 2 0 1 0.406 0.370 0.100 0.470 0 -0.470 0.000 -0.094 -0.094 2 1 0 0.406 0.276 0.006 0.412 0 -0.412 -0.082 0.000 -0.082 2 1 1 0.323 0.276 -0.076 0.523 1 0.477 0.095 0.095 0.095 … … 89 0 0 0.625 0.562 -0.312 -0.312 0 0.312 0 0 0.062 89 0 1 0.625 0.562 -0.25 0.313 0 -0.313 0 -0.063 -0.063 89 1 0 0.625 0.500 -0.312 0.313 0 -0.313 -0.063 0 -0.063 89 1 1 0.562 0.500 -0.375 0.687 1 0.313 0.063 0.063 0.063 After 89 epochs, we find the network to produce values near to the desired output. Since in this example the outputs are binary (zero or one), we can assume that any value produced by the network that is below 0.5 is considered to be 0 and any value above 0.5 is considered to be 1. So, we can draw a function , with the final weights and bias found by the learning algorithm w1=0.562, w2=0.5 and b=-0.375, defining the linear boundary in the chart: This boundary is a definition of all classifications given by the network. You can see that the boundary is linear, given that the function is also linear. Thus, the perceptron network is really suitable for problems whose patterns are linearly separable. The XOR case Now let's analyze the XOR case: We see that in two dimensions, it is impossible to draw a line to separate the two patterns. What would happen if we tried to train a single layer perceptron to learn this function? Suppose we tried, let's see what happened in the following table: Epoch x1 x2 w1 w2 b y t E Δw1 Δw2 Δb 1 0 0 0.5 0.5 0.5 0.5 0 -0.5 0 0 -0.1 1 0 1 0.5 0.5 0.4 0.9 1 0.1 0 0.02 0.02 1 1 0 0.5 0.52 0.42 0.92 1 0.08 0.016 0 0.016 1 1 1 0.516 0.52 0.436 1.472 0 -1.472 -0.294 -0.294 -0.294 2 0 0 0.222 0.226 0.142 0.142 0 -0.142 0.000 0.000 -0.028 2 0 1 0.222 0.226 0.113 0.339 1 0.661 0.000 0.132 0.132 2 1 0 0.222 0.358 0.246 0.467 1 0.533 0.107 0.000 0.107 2 1 1 0.328 0.358 0.352 1.038 0 -1.038 -0.208 -0.208 -0.208 … … 127 0 0 -0.250 -0.125 0.625 0.625 0 -0.625 0.000 0.000 -0.125 127 0 1 -0.250 -0.125 0.500 0.375 1 0.625 0.000 0.125 0.125 127 1 0 -0.250 0.000 0.625 0.375 1 0.625 0.125 0.000 0.125 127 1 1 -0.125 0.000 0.750 0.625 0 -0.625 -0.125 -0.125 -0.125 The perceptron just could not find any pair of weights that would drive the following error 0.625. This can be explained mathematically as we already perceived from the chart that this function cannot be linearly separable in two dimensions. So what if we add another dimension? Let's see the chart in three dimensions: In three dimensions, it is possible to draw a plane that would separate the patterns, provided that this additional dimension could properly transform the input data. Okay, but now there is an additional problem: how could we derive this additional dimension since we have only two input variables? One obvious, but also workaround, answer would be adding a third variable as a derivation from the two original ones. And being this third variable a (derivation), our neural network would probably get the following shape: Okay, now the perceptron has three inputs, one of them being a composition of the other. This also leads to a new question: how should that composition be processed? We can see that this component could act as a neuron, so giving the neural network a nested architecture. If so, there would another new question: how would the weights of this new neuron be trained, since the error is on the output neuron? Multi-layer perceptrons As we can see, one simple example in which the patterns are not linearly separable has led us to more and more issue using the perceptron architecture. That need led to the application of multilayer perceptrons. The fact that the natural neural network is structured in layers as well, and each layer captures pieces of information from a specific environment is already established. In artificial neural networks, layers of neurons act in this way, by extracting and abstracting information from data, transforming them into another dimension or shape. In the XOR example, we found the solution to be the addition of a third component that would make possible a linear separation. But there remained a few questions regarding how that third component would be computed. Now let's consider the same solution as a two-layer perceptron: Now we have three neurons instead of just one, but in the output the information transferred by the previous layer is transformed into another dimension or shape, whereby it would be theoretically possible to establish a linear boundary on those data points. However, the question on finding the weights for the first layer remains unanswered, or can we apply the same training rule to neurons other than the output? We are going to deal with this issue in the Generalized delta rule section. MLP properties Multi-layer perceptrons can have any number of layers and also any number of neurons in each layer. The activation functions may be different on any layer. An MLP network is usually composed of at least two layers, one for the output and one hidden layer. There are also some references that consider the input layer as the nodes that collect input data; therefore, for those cases, the MLP is considered to have at least three layers. For the purpose of this article, let's consider the input layer as a special type of layer which has no weights, and as the effective layers, that is, those enabled to be trained, we'll consider the hidden and output layers. A hidden layer is called that because it actually hides its outputs from the external world. Hidden layers can be connected in series in any number, thus forming a deep neural network. However, the more layers a neural network has, the slower would be both training and running, and according to mathematical foundations, a neural network with one or two hidden layers at most may learn as well as deep neural networks with dozens of hidden layers. But it depends on several factors. MLP weights In an MLP feedforward network, one particular neuron i receives data from a neuron j of the previous layer and forwards its output to a neuron k of the next layer: The mathematical description of a neural network is recursive: Here, yo is the network output (should we have multiple outputs, we can replace yo with Y, representing a vector); fo is the activation function of the output; l is the number of hidden layers; nhi is the number of neurons in the hidden layer i; wi is the weight connecting the i th neuron of the last hidden layer to the output; fi is the activation function of the neuron i; and bi is the bias of the neuron i. It can be seen that this equation gets larger as the number of layers increases. In the last summing operation, there will be the inputs xi. Recurrent MLP The neurons on an MLP may feed signals not only to neurons in the next layers (feedforward network), but also to neurons in the same or previous layers (feedback or recurrent). This behavior allows the neural network to maintain state on some data sequence, and this feature is especially exploited when dealing with time series or handwriting recognition. Recurrent networks are usually harder to train, and eventually the computer may run out of memory while executing them. In addition, there are recurrent network architectures better than MLPs, such as Elman, Hopfield, Echo state, Bidirectional RNNs (recurrent neural networks). But we are not going to dive deep into these architectures. Coding an MLP Bringing these concepts into the OOP point of view, we can review the classes already designed so far: One can see that the neural network structure is hierarchical. A neural network is composed of layers that are composed of neurons. In the MLP architecture, there are three types of layers: input, hidden, and output. So suppose that in Java, we would like to define a neural network consisting of three inputs, one output (linear activation function) and one hidden layer (sigmoid function) containing five neurons. The resulting code would be as follows: int numberOfInputs=3; int numberOfOutputs=1; int[] numberOfHiddenNeurons={5};     Linear outputAcFnc = new Linear(1.0); Sigmoid hiddenAcFnc = new Sigmoid(1.0); NeuralNet neuralnet = new NeuralNet(numberOfInputs, numberOfOutputs, numberOfHiddenNeurons, hiddenAcFnc, outputAcFnc); To summarize, we saw how perceptrons can be applied to solve linear separation problems, their limitations in classifying nonlinear data and how to suppress those limitations with multi-layer perceptrons (MLPs). If you enjoyed this excerpt, check out the book Neural Network Programming with Java Second Edition for a better understanding of neural networks and how they fit in different real-world projects.
Read more
  • 0
  • 0
  • 23853

article-image-c-sfml-visual-studio-and-starting-first-game
Packt
19 Aug 2016
15 min read
Save for later

C++, SFML, Visual Studio, and Starting the first game

Packt
19 Aug 2016
15 min read
In this article by John Horton, author of the book, Beginning C++ Game Programming, we will waste no time in getting you started on your journey to writing great games for the PC, using C++ and the OpenGL powered SFML.We will learn absolutely everything we need, to have the first part of our first game up and running. Here is what we will do now. (For more resources related to this topic, see here.) Find out about the games we will build Learn a bit about C++ Explore SFML and its relationship with C++ Look at the software, Visual Studio Setup a game development environment Create a reusable project template which will save a lot of time Plan and prepare for the first game project Timber!!! Write the first C++ code and make a runnable game that draws a background The games The journey will be mainly smooth as we will learn the fundamentals of the super-fast C++ language, a step at a time and then put the new knowledge to use, adding cool features to the three games we are building. Timber!!! The first game is an addictive, fast-paced clone of the hugely successful Timberman http://store.steampowered.com/app/398710/. Our game, Timber!!!, will allow us to be introduced to all the C++ basics at the same time as building a genuinely playable game. Here is what our version of the game will look like when we are done and we have added a few last-minute enhancements. The Walking Zed Next we will build a frantic, zombie survival-shooter, not unlike the Steam hit, Over 9000 Zombies http://store.steampowered.com/app/273500/. The player will have a machine gun, the ability to gather resources and build defenses. All this will take place in a randomly generated, scrolling world. To achieve this we will learn about object oriented programming and how it enables us to have a large code base (lots of code) that is easy to write and maintain. Expect exciting features like hundreds of enemies, rapid fire weaponry and directional sound. Thomas gets a real friend The third game will be a stylish and challenging, single player and coop, puzzle platformer. It is based on the very popular game, Thomas was Alone http://store.steampowered.com/app/220780/. Expect to learn cool topics like particle effects, OpenGL Shaders and multiplayer networking. If you want to play any of the games now, you can do so from the download bundle in the Runnable Games folder. Just double-click on the appropriate .exe file. Let's get started by introducing C++, Visual Studio and SFML! C++ One question you might have is, why use C++ at all? C++ is fast, very fast. What makes this so, is the fact that the code that we write is directly translated into machine executable instructions. These instructions together, are what makes the game. The executable game is contained within a .exe file which the player can simply double-click to run. There are a few steps in the process. First the pre-processor looks to see if any other code needs to be included within our own and adds it if necessary. Next, all the code is compiled into object files by the compiler program. Finally a third program called the linker, joins all the object files into the executable file which is our game. In addition, C++ is well established at the same time as being extremely up-to-date. C++ is an Object Oriented Programming (OOP) language which means we can write and organize our code in a proven way that makes our games efficient and manageable. Most of this other code that I refered to, you might be able to guess, is SFML, and we will find out more about SFML in just a minute. The pre-processor, compiler and linker programs I have just mentioned, are all part of the Visual Studio Integrated Development Environment (IDE). Microsoft Visual Studio Visual Studio hides away the complexity of the pre-processing, compiling and linking. It wraps it all up into the press of a button. In addition to this, it provides a slick user interface for us to type our code and manage, what will become a large selection of code files and other project assets as well. While there are advanced versions of Visual Studio that cost hundreds of dollars, we will be able to build all three of our games in the free Express 2015 for Desktop version. SFML Simple Fast Media Library (SFML) is not the only C++ library for games. It is possible to make an argument to use other libraries, but SFML seems to come through in first place, for me, every time. Firstly it is written using object oriented C++. Perhaps the biggest benefit is that all modern C++ programming uses OOP. Every C++ beginners guide I have ever read. uses and teaches OOP. OOP is the future (and the now) of coding in almost all languages in fact. So why, if your learning C++ from the beginning, would you want to do it any other way? SFML has a module (code) for just about anything you would ever want to do in a 2d game. SFML works using OpenGL that can also make 3d games. OpenGL is the de-facto free-to-use graphics library for games. When you use SFML, you are automatically using OpenGL. SFML drastically simplifies: 2d graphics and animation including scrolling game-worlds. Sound effects and music playback, including high quality directional sound. Online multiplayer features The same code can be compiled and linked on all major desktop operating systems, and soon mobile as well! Extensive research has not uncovered any more suitable way to build 2d games for PC, even for expert developers and especially if you are a beginner and want to learn C++ in a fun gaming environment. Setting up the development environment Now you know a bit more about how we will be making games, it is time to set up a development environment so we can get coding. What about Mac and Linux? The games that we make can be built to run on Windows, Mac and Linux! The code we use will be identical for each each. However, each version does need to be compiled and linked on the platform for which it is intended and Visual Studio will not be able to help us with Mac and Linux. Although, I guess, if you are an enthusiastic Mac or Linux user and you are comfortable with your operating system, the vast majority of the challenge you will encounter, will be in the initial setup of the development environment, SFML and the first project. For Linux, read this for an overview: http://www.sfml-dev.org/tutorials/2.0/start-linux.php For Linux, read this for step-by-step: http://en.sfml-dev.org/forums/index.php?topic=9808.0 On Mac, read this tutorial as well as the linked out articles: http://www.edparrish.net/common/sfml-osx.html Installing Visual Studio Express 2015 for Desktop Installing Visual Studio can be almost as simple as downloading a file and clicking a few buttons. It will help us, however, if we carefully run through exactly how we do this. For this reason I will walk through the installation process a step at a time. The Microsoft Visual Studio site says, you need 5 GB of hard disk space. From experience, however, I would suggest you need at least 10 GB of free space. In addition, these figures are slightly ambiguous. If you are planning to install on a secondary hard drive, you will still need at least 5 GB on the primary hard drive because no matter where you choose to install Visual Studio, it will need this space too. To summarize this ambiguous situation: It is essential to have a full 10 GB space on the primary hard disk, if you will be installing Visual Studio to that primary hard disk. On the other hand, make sure you have 5 GB on the primary hard disk as well as 10 GB on the secondary, if you intend to install to a secondary hard disk. Yep, stupid, I know! The first thing you need is a Microsoft account and the login details. If you have a Hotmail or MSN email address then you already have one. If not, you can sign up for a free one here:https://login.live.com/. Visit this link: https://www.visualstudio.com/en-us/downloads/download-visual-studio-vs.aspx. Click on Visual Studio 2015, then Express 2015 for desktop then the Download button. This next image shows the three places to click. Wait for the short download to complete and then run the downloaded file. Now you just need to follow the on-screen instructions. However, make a note of the folder where you choose to install Visual Studio. If you want to do things exactly the same as me, then create a new folder called Visual Studio 2015 on your preferred hard disk and install to this folder. This whole process could take a while depending on the speed of your Internet connection. When you see the next screen, click on Launch and enter your Microsoft account login details. Now we can turn to SFML. Setting up SFML This short tutorial will step through downloading the SFML files, that allows us to include the functionality contained in the library as well as the files, called DLL files, that will enable us to link our compiled object code to the SFML compiled object code. Visit this link on the SFML website: http://www.sfml-dev.org/download.php. Click on the button that says Latest Stable Version as shown next. By the time you read this guide, the actual latest version will almost certainly have changed. That doesn't matter as long as you do the next step just right. We want to download the 32 bit version for Visual C++ 2014. This might sound counter-intuitive because we have just installed Visual Studio 2015 and you probably(most commonly) have a 64 bit PC. The reason we choose the download that we do, is because Visual C++ 2014 is part of Visual Studio 2015 (Visual Studio does more than C++) and we will be building games in 32 bit so they run on both 32 and 64 bit machines. To be clear click the download indicated below. When the download completes, create a folder at the root of the same drive where you installed Visual Studio and name it SFML. Also create another folder at the root of the drive where you installed Visual Studio and call it Visual Studio Stuff. We will store all kinds of Visual Studio related things here so Stuff seems like a good name. Just to be clear, here is what my hard drive looks like after this step Obviously, the folders you have in between the highlighted three folders in the image will probably be totally different to mine. Now, ready for all the projects we will soon be making, create a new folder inside Visual Studio Stuff. Name the new folder Projects. Finally, unzip the SFML download. Do this on your desktop. When unzipping is complete you can delete the zip folder. You will be left with a single folder on your desktop. Its name will reflect the version of SFML that you downloaded. Mine is called SFML-2.3.2-windows-vc14-32-bit. Your file name will likely reflect a more recent version. Double click this folder to see the contents, then double click again into the next folder (mine is called SFML-2.3.2). The image below is what my SFML-2.3.2 folder contents looks like, when the entire contents has been selected. Yours should look the same. Copy the entire contents of this folder, as seen in the previous image and paste/drag all the contents into the SFML folder you created in step 3. I will refer to this folder simply as your SFML folder. Now we are ready to start using C++ and SFML in Visual Studio. Creating a reusable project template As setting up a project is a fairly fiddly process, we will create a project and then save it as a Visual Studio template. This will save us a quite significant amount of work each time we start a new game. So if you find the next tutorial a little tedious, rest assured that you will never need to do this again. In the New Project window, click the little drop-down arrow next to Visual C++ to reveal more options, then click Win32 and then click Win32 Console Application, You can see all these selections in the next screen-shot. Now, at the bottom of the New Project window type HelloSFML in the Name: field. Next, browse to the Visual Studio StuffProjects folder that we created in the previous tutorial. This will be the location that all our project files will be kept. All templates are based on an actual project. So we will have a project called HelloSFML but the only thing we will do with it, is make a template from it. When you have completed the steps above click OK. The next image shows the Application Settings window. Check the box for Console application, and leave the other options as shown below. Click Finish and Visual Studio will create the new project. Next we will add some fairly intricate and important project settings. This is the laborious part, but as we will create a template, we will only need to do this once. What we need to do is to tell Visual Studio, or more specifically the code compiler, that is part of Visual Studio, where to find a special type of code file from SFML. The special type of file I am referring to is a header file. Header files are the files that define the format of the SFML code. So when we use the SFML code, the compiler knows how to handle it. Note that the header files are distinct from the main source code files and they are contained in files with the .hpp file extension. All this will become clearer when we eventually start adding our own header files in the second project. In addition, we need to tell Visual Studio where it can find the SFML library files. From the Visual Studio main menu select Project | HelloSFML properties. In the resulting HelloSFML Property Pages window, take the following steps, which are numbered and can be referred to in the next image. First, select All Configurations from the Configuration: drop-down. Second, select C/C++ then General from the left-hand menu. Third, locate the Additional Include Directories edit box and type the drive letter where your SFML folder is located, followed by SFMLinclude. The full path to type, if you located your SFML folder on your D drive is, as shown in the screen-shot, D:SFMLinclude. Vary your path if you installed SFML to a different drive. Click Apply to save your configurations so far. Now, still in the same window, perform these next steps which again refer to the next image. Select Linker then General. Find the Additional Library Directories edit box and type the drive letter where your SFML folder is, followed by SFMLlib. So the full path to type if you located your SFML folder on your D drive is, as shown in the screen-shot, D:SFMLlib. Vary your path if you installed SFML to a different drive. Click Apply to save your configurations so far. Finally for this stage, still in the same window, perform these steps which again refer to the next image. Switch the Configuration: drop down(1) to Debug as we will be running and testing our games in debug mode. Select Linker then Input (2). Find the Additional Dependencies edit box (3) and click into it at the far left hand side. Now copy & paste/type the following at the indicated place.: sfml-graphics-d.lib;sfml-window-d.lib;sfml-system-d.lib;sfml-network-d.lib;sfml-audio-d.lib; Again be REALLY careful to place the cursor exactly and not to overwrite any of the text that is already there. Click OK. Let's make the template of our HelloSFML project so we never have to do this slightly mind-numbing task again. Creating a reusable project template is really easy. In Visual Studio select File | Export Template…. Then in the Export Template Wizard window make sure the Project template option is selected and the HelloSFML project is selected for the From which project do you want to create a template option. Click Next and then Finish. Phew, that's it! Next time we create a project I'll show you how to do it from this template. Let's build Timber!!! Summary In this article we learnt that, it is true that configuring an IDE, to use a C++ library can be a bit awkward and long. Also the concept of classes and objects is well known to be slightly awkward for people new to coding. Resources for Article: Further resources on this subject: Game Development Using C++ [Article] Connecting to Microsoft SQL Server Compact 3.5 with Visual Studio [Article] Introducing the Boost C++ Libraries [Article]
Read more
  • 0
  • 0
  • 23838

article-image-apollo-11-source-code-how-it-became-a-small-step-for-a-woman-and-a-huge-leap-for-software-engineering
Sugandha Lahoti
19 Jul 2018
5 min read
Save for later

Apollo 11 source code: A small step for a woman, and a huge leap for 'software engineering'

Sugandha Lahoti
19 Jul 2018
5 min read
Yesterday, reddit saw an explosion of discussion around the original Apollo 11 Guidance Computer (AGC) source code. The code in its entirety was uploaded on GitHub two years ago, thanks to former NASA intern, Chris Garry. And again it seems to have undergone significant updates this week looking at the timestamps on all the files in the repo. This is a project that will always hold a special place for all software professionals around the world. This is the project that made ‘software engineering’ a real discipline. What is AGC and why it mattered for Apollo 11? AGC was a digital computer produced for the Apollo program, installed on board the Apollo 11 Command Module (CM) and Lunar Module (LM). The AGC code is also referred to as ‘COLOSSUS 2A’ and was written in AGC assembly language and stored on rope memory. On any given Apollo mission, there were two AGCs, one for the CM, and one for the LM. The two AGCs were identical and interchangeable. However, their software differed as both the LM and the CM performed different tasks pertaining to the spacecraft. The CM launched the three astronauts to the moon, and back again. The LM helped in the landing of two of the astronauts on the moon while the third astronaut remained in the CM, in orbit around the moon. The woman who coined the term, ‘software engineering’ The AGC code was brought to life by Margaret Hamilton, director of software engineering for the project. In a male-dominated world of tech and engineering of that time, Margaret was an exception. She led a team credited with developing the software for Apollo and Skylab, keeping her head high even through backlash. “People used to say to me, ‘How can you leave your daughter? How can you do this?”  She went on to become the founder and CEO of Hamilton Technologies, Inc. and was also awarded the Presidential Medal of Freedom in 2016. Hamilton is considered one of the pioneers of software engineering, credited for actually coining the term “software engineering”. She first started using the term during the early Apollo missions wanting to give software the same legitimacy as other disciplines. At that time, it was not taken seriously but over time software engineering has become an IEEE Standard. What can we learn from the AGC code developers? Understandably, the AGC specifications and processing power are very underrated as compared to the technology of today. Some still wittingly call it a calculator, instead of a computer. Others say, that the CPU in a microwave oven is probably more powerful than an AGC. Inspite of being a very basic technology in terms of processing power and speed, the Apollo 11 spacecraft was able to complete the first ever manned mission to the moon and back. This is not just a huge testament to the original programming team’s ingenuity and resourcefulness but also of their grit and meticulousness. One would think such a bunch produced serious (boring) code that has flawless execution. Read between the (code) lines and you see a team that just enjoyed every moment of writing the code with quirky naming conventions and humorous notes inside the comments. Back to the present As soon as the code was uploaded on Github two years ago and even now, coders and software programmers all over the world, are dissecting it, particularly interested in the quirky English-descriptions of code explanations. People on Reddit are terming the code files as real programming that doesn’t rely on APIs to do the heavy lifting. People are also loving the naming convention of the source code files and their programs which were 1960s inspired light-hearted jokes. For example, the BURN_BABY_BURN--MASTER IGNITION ROUTINE for the master ignition routine, and the PINBALL_GAME_BUTTONS_AND_LIGHTS.agc for keyboard and display code. Even the programs are quirky. The LUNAR_LANDING_GUIDANCE_EQUATIONS.s, file ended up having two temporary lines of code as permanent. You can read more such interesting reddit comments. However, a point worth noting is that Margaret and by extension, women in tech, are conspicuously missing in this rich discussion. We can start seeing real change only when discussion forums start including various facets of the tool/tech under discussion. People behind the tech are an important facet, and more so when they are in the minority. You can also read The Apollo Guidance Computer: Architecture and Operation the inside scoop on how the AGC functioned, what kind of design decisions and software choices the programmers had to made based on the features and limitations of the AGC among other insights. The github repo for the original Apollo 11 source code also contains material for further reading. Is space the final frontier for AI? NASA to reveal Kepler’s latest AI backed discovery NASA’s Kepler discovers a new exoplanet using Google’s Machine Learning Meet CIMON, the first AI robot to join the astronauts aboard ISS
Read more
  • 0
  • 1
  • 23827

article-image-putting-function-functional-programming
Packt
22 Sep 2015
27 min read
Save for later

Putting the Function in Functional Programming

Packt
22 Sep 2015
27 min read
 In this article by Richard Reese, the author of the book Learning Java Functional Programming, we will cover lambda expressions in more depth. We will explain how they satisfy the mathematical definition of a function and how we can use them in supporting Java applications. In this article, you will cover several topics, including: Lambda expression syntax and type inference High-order, pure, and first-class functions Referential transparency Closure and currying (For more resources related to this topic, see here.) Our discussions cover high-order functions, first-class functions, and pure functions. Also examined are the concepts of referential transparency, closure, and currying. Examples of nonfunctional approaches are followed by their functional equivalent where practical. Lambda expressions usage A lambda expression can be used in many different situations, including: Assigned to a variable Passed as a parameter Returned from a function or method We will demonstrate how each of these are accomplished and then elaborate on the use of functional interfaces. Consider the forEach method supported by several classes and interfaces, including the List interface. In the following example, a List interface is created and the forEach method is executed against it. The forEach method expects an object that implements the Consumer interface. This will display the three cartoon character names: List<String> list = Arrays.asList("Huey", "Duey", "Luey"); list.forEach(/* Implementation of Consumer Interface*/); More specifically, the forEach method expects an object that implements the accept method, the interface's single abstract method. This method's signature is as follows: void accept(T t) The interface also has a default method, andThen, which is passed and returns an instance of the Consumer interface. We can use any of three different approaches for implementing the functionality of the accept method: Use an instance of a class that implements the Consumer interface Use an anonymous inner class Use a lambda expression We will demonstrate each method so that it will be clear how each technique works and why lambda expressions will often result in a better solution. We will start with the declaration of a class that implements the Consumer interface as shown next: public class ConsumerImpl<T> implements Consumer<T> { @Override public void accept(T t) { System.out.println(t); } } We can then use it as the argument of the forEach method: list.forEach(new ConsumerImpl<>()); Using an explicit class allows us to reuse the class or its objects whenever an instance is needed. The second approach uses an anonymous inner function as shown here: list.forEach(new Consumer<String>() { @Override public void accept(String t) { System.out.println(t); } }); This was a fairly common approach used prior to Java 8. It avoids having to explicitly declare and instantiate a class, which implements the Consumer interface. A simple statement that uses a lambda expression is shown next: list.forEach(t->System.out.println(t)); The lambda expression accepts a single argument and returns void. This matches the signature of the Consumer interface. Java 8 is able to automatically perform this matching process. This latter technique obviously uses less code, making it more succinct than the other solutions. If we desire to reuse this lambda expression elsewhere, we could have assigned it to a variable first and then used it in the forEach method as shown here: Consumer consumer = t->System.out.println(t); list.forEach(consumer); Anywhere a functional interface is expected, we can use a lambda expression. Thus, the availability of a large number of functional interfaces will enable the frequent use of lambda expressions and programs that exhibit a functional style of programming. While developers can define their own functional interfaces, which we will do shortly, Java 8 has added a large number of functional interfaces designed to support common operations. Most of these are found in the java.util.function package. We will use several of these throughout the book and will elaborate on their purpose, definition, and use as we encounter them. Functional programming concepts in Java In this section, we will examine the underlying concept of functions and how they are implemented in Java 8. This includes high-order, first-class, and pure functions. A first-class function is a function that can be used where other first-class entities can be used. These types of entities include primitive data types and objects. Typically, they can be passed to and returned from functions and methods. In addition, they can be assigned to variables. A high-order function either takes another function as an argument or returns a function as the return value. Languages that support this type of function are more flexible. They allow a more natural flow and composition of operations. Pure functions have no side effects. The function does not modify nonlocal variables and does not perform I/O. High-order functions We will demonstrate the creation and use of the high-order function using an imperative and a functional approach to convert letters of a string to lowercase. The next code sequence reuses the list variable, developed in the previous section, to illustrate the imperative approach. The for-each statement iterates through each element of the list using the String class' toLowerCase method to perform the conversion: for(String element : list) { System.out.println(element.toLowerCase()); } The output will be each name in the list displayed in lowercase, each on a separate line. To demonstrate the use of a high-order function, we will create a function called, processString, which is passed a function as the first parameter and then apply this function to the second parameter as shown next:   public String processString(Function<String,String> operation,String target) { return operation.apply(target); } The function passed will be an instance of the java.util.function package's Function interface. This interface possesses an accept method that is passed one data type and returns a potentially different data type. With our definition, it is passed String and returns String. In the next code sequence, a lambda expression using the toLowerCase method is passed to the processString method. As you may remember, the forEach method accepts a lambda expression, which matches the Consumer interface's accept method. The lambda expression passed to the processString method matches the Function interface's accept method. The output is the same as produced by the equivalent imperative implementation. list.forEach(s ->System.out.println( processString(t->t.toLowerCase(), s))); We could have also used a method reference as show next: list.forEach(s ->System.out.println( processString(String::toLowerCase, s))); The use of the high-order function may initially seem to be a bit convoluted. We needed to create the processString function and then pass either a lambda expression or a method reference to perform the conversion. While this is true, the benefit of this approach is flexibility. If we needed to perform a different string operation other than converting the target string to lowercase, we will need to essentially duplicate the imperative code and replace toLowerCase with a new method such as toUpperCase. However, with the functional approach, all we need to do is replace the method used as shown next: list.forEach(s ->System.out.println(processString(t- >t.toUpperCase(), s))); This is simpler and more flexible. A lambda expression can also be passed to another lambda expression. Let's consider another example where high-order functions can be useful. Suppose we need to convert a list of one type into a list of a different type. We might have a list of strings that we wish to convert to their integer equivalents. We might want to perform a simple conversion or perhaps we might want to double the integer value. We will use the following lists:   List<String> numberString = Arrays.asList("12", "34", "82"); List<Integer> numbers = new ArrayList<>(); List<Integer> doubleNumbers = new ArrayList<>(); The following code sequence uses an iterative approach to convert the string list into an integer list:   for (String num : numberString) { numbers.add(Integer.parseInt(num)); } The next sequence uses a stream to perform the same conversion: numbers.clear(); numberString .stream() .forEach(s -> numbers.add(Integer.parseInt(s))); There is not a lot of difference between these two approaches, at least from a number of lines perspective. However, the iterative solution will only work for the two lists: numberString and numbers. To avoid this, we could have written the conversion routine as a method. We could also use lambda expression to perform the same conversion. The following two lambda expression will convert a string list to an integer list and from a string list to an integer list where the integer has been doubled:   Function<List<String>, List<Integer>> singleFunction = s -> { s.stream() .forEach(t -> numbers.add(Integer.parseInt(t))); return numbers; }; Function<List<String>, List<Integer>> doubleFunction = s -> { s.stream() .forEach(t -> doubleNumbers.add( Integer.parseInt(t) * 2)); return doubleNumbers; }; We can apply these two functions as shown here: numbers.clear(); System.out.println(singleFunction.apply(numberString)); System.out.println(doubleFunction.apply(numberString)); The output follows: [12, 34, 82] [24, 68, 164] However, the real power comes from passing these functions to other functions. In the next code sequence, a stream is created consisting of a single element, a list. This list contains a single element, the numberString list. The map method expects a Function interface instance. Here, we use the doubleFunction function. The list of strings is converted to integers and then doubled. The resulting list is displayed: Arrays.asList(numberString).stream() .map(doubleFunction) .forEach(s -> System.out.println(s)); The output follows: [24, 68, 164] We passed a function to a method. We could easily pass other functions to achieve different outputs. Returning a function When a value is returned from a function or method, it is intended to be used elsewhere in the application. Sometimes, the return value is used to determine how subsequent computations should proceed. To illustrate how returning a function can be useful, let's consider a problem where we need to calculate the pay of an employee based on the numbers of hours worked, the pay rate, and the employee type. To facilitate the example, start with an enumeration representing the employee type: enum EmployeeType {Hourly, Salary, Sales}; The next method illustrates one way of calculating the pay using an imperative approach. A more complex set of computation could be used, but these will suffice for our needs: public float calculatePay(int hoursWorked, float payRate, EmployeeType type) { switch (type) { case Hourly: return hoursWorked * payRate; case Salary: return 40 * payRate; case Sales: return 500.0f + 0.15f * payRate; default: return 0.0f; } } If we assume a 7 day workweek, then the next code sequence shows an imperative way of calculating the total number of hours worked: int hoursWorked[] = {8, 12, 8, 6, 6, 5, 6, 0}; int totalHoursWorked = 0; for (int hour : hoursWorked) { totalHoursWorked += hour; } Alternatively, we could have used a stream to perform the same operation as shown next. The Arrays class's stream method accepts an array of integers and converts it into a Stream object. The sum method is applied fluently, returning the number of hours worked: totalHoursWorked = Arrays.stream(hoursWorked).sum(); The latter approach is simpler and easier to read. To calculate and display the pay, we can use the following statement which, when executed, will return 803.25.    System.out.println( calculatePay(totalHoursWorked, 15.75f, EmployeeType.Hourly)); The functional approach is shown next. A calculatePayFunction method is created that is passed by the employee type and returns a lambda expression. This will compute the pay based on the number of hours worked and the pay rate. This lambda expression is based on the BiFunction interface. It has an accept method that takes two arguments and returns a value. Each of the parameters and the return type can be of different data types. It is similar to the Function interface's accept method, except that it is passed two arguments instead of one. The calculatePayFunction method is shown next. It is similar to the imperative's calculatePay method, but returns a lambda expression: public BiFunction<Integer, Float, Float> calculatePayFunction( EmployeeType type) { switch (type) { case Hourly: return (hours, payRate) -> hours * payRate; case Salary: return (hours, payRate) -> 40 * payRate; case Sales: return (hours, payRate) -> 500f + 0.15f * payRate; default: return null; } } It can be invoked as shown next: System.out.println( calculatePayFunction(EmployeeType.Hourly) .apply(totalHoursWorked, 15.75f)); When executed, it will produce the same output as the imperative solution. The advantage of this approach is that the lambda expression can be passed around and executed in different contexts. First-class functions To demonstrate first-class functions, we use lambda expressions. Assigning a lambda expression, or method reference, to a variable can be done in Java 8. Simply declare a variable of the appropriate function type and use the assignment operator to do the assignment. In the following statement, a reference variable to the previously defined BiFunction-based lambda expression is declared along with the number of hours worked: BiFunction<Integer, Float, Float> calculateFunction; int hoursWorked = 51; We can easily assign a lambda expression to this variable. Here, we use the lambda expression returned from the calculatePayFunction method: calculateFunction = calculatePayFunction(EmployeeType.Hourly); The reference variable can then be used as shown in this statement: System.out.println( calculateFunction.apply(hoursWorked, 15.75f)); It produces the same output as before. One shortcoming of the way an hourly employee's pay is computed is that overtime pay is not handled. We can add this functionality to the calculatePayFunction method. However, to further illustrate the use of reference variables, we will assign one of two lambda expressions to the calculateFunction variable based on the number of hours worked as shown here: if(hoursWorked<=40) { calculateFunction = (hours, payRate) -> 40 * payRate; } else { calculateFunction = (hours, payRate) -> hours*payRate + (hours-40)*1.5f*payRate; } When the expression is evaluated as shown next, it returns a value of 1063.125: System.out.println( calculateFunction.apply(hoursWorked, 15.75f)); Let's rework the example developed in the High-order functions section, where we used lambda expressions to display the lowercase values of an array of string. Part of the code has been duplicated here for your convenience: list.forEach(s ->System.out.println( processString(t->t.toLowerCase(), s))); Instead, we will use variables to hold the lambda expressions for the Consumer and Function interfaces as shown here: Consumer<String> consumer; consumer = s -> System.out.println(toLowerFunction.apply(s)); Function<String,String> toLowerFunction; toLowerFunction= t -> t.toLowerCase(); The declaration and initialization could have been done with one statement for each variable. To display all of the names, we simply use the consumer variable as the argument of the forEach method: list.forEach(consumer); This will display the names as before. However, this is much easier to read and follow. The ability to use lambda expressions as first-class entities makes this possible. We can also assign method references to variables. Here, we replaced the initialization of the function variable with a method reference: function = String::toLowerCase; The output of the code will not change. The pure function The pure function is a function that has no side effects. By side effects, we mean that the function does not modify nonlocal variables and does not perform I/O. A method that squares a number is an example of a pure method with no side effects as shown here: public class SimpleMath { public static int square(int x) { return x * x; } } Its use is shown here and will display the result, 25: System.out.println(SimpleMath.square(5)); An equivalent lambda expression is shown here: Function<Integer,Integer> squareFunction = x -> x*x; System.out.println(squareFunction.apply(5)); The advantages of pure functions include the following: They can be invoked repeatedly producing the same results There are no dependencies between functions that impact the order they can be executed They support lazy evaluation They support referential transparency We will examine each of these advantages in more depth. Support repeated execution Using the same arguments will produce the same results. The previous square operation is an example of this. Since the operation does not depend on other external values, re-executing the code with the same arguments will return the same results. This supports the optimization technique call memoization. This is the process of caching the results of an expensive execution sequence and retrieving them when they are used again. An imperative technique for implementing this approach involves using a hash map to store values that have already been computed and retrieving them when they are used again. Let's demonstrate this using the square function. The technique should be used for those functions that are compute intensive. However, using the square function will allow us to focus on the technique. Declare a cache to hold the previously computed values as shown here: private final Map<Integer, Integer> memoizationCache = new HashMap<>(); We need to declare two methods. The first method, called doComputeExpensiveSquare, does the actual computation as shown here. A display statement is included only to verify the correct operation of the technique. Otherwise, it is not needed. The method should only be called once for each unique value passed to it. private Integer doComputeExpensiveSquare(Integer input) { System.out.println("Computing square"); return 2 * input; } A second method is used to detect when a value is used a subsequent time and return the previously computed value instead of calling the square method. This is shown next. The containsKey method checks to see if the input value has already been used. If it hasn't, then the doComputeExpensiveSquare method is called. Otherwise, the cached value is returned. public Integer computeExpensiveSquare(Integer input) { if (!memoizationCache.containsKey(input)) { memoizationCache.put(input, doComputeExpensiveSquare(input)); } return memoizationCache.get(input); } The use of the technique is demonstrated with the next code sequence: System.out.println(computeExpensiveSquare(4)); System.out.println(computeExpensiveSquare(4)); The output follows, which demonstrates that the square method was only called once: Computing square 16 16 The problem with this approach is the declaration of a hash map. This object may be inadvertently used by other elements of the program and will require the explicit declaration of new hash maps for each memoization usage. In addition, it does not offer flexibility in handling multiple memoization. A better approach is available in Java 8. This new approach wraps the hash map in a class and allows easier creation and use of memoization. Let's examine a memoization class as adapted from http://java.dzone.com/articles/java-8-automatic-memoization. It is called Memoizer. It uses ConcurrentHashMap to cache value and supports concurrent access from multiple threads. Two methods are defined. The doMemoize method returns a lambda expression that does all of the work. The memorize method creates an instance of the Memoizer class and passes the lambda expression implementing the expensive operation to the doMemoize method. The doMemoize method uses the ConcurrentHashMap class's computeIfAbsent method to determine if the computation has already been performed. If the value has not been computed, it executes the Function interface's apply method against the function argument: public class Memoizer<T, U> { private final Map<T, U> memoizationCache = new ConcurrentHashMap<>(); private Function<T, U> doMemoize(final Function<T, U> function) { return input -> memoizationCache.computeIfAbsent(input, function::apply); } public static <T, U> Function<T, U> memoize(final Function<T, U> function) { return new Memoizer<T, U>().doMemoize(function); } } A lambda expression is created for the square operation: Function<Integer, Integer> squareFunction = x -> { System.out.println("In function"); return x * x; }; The memoizationFunction variable will hold the lambda expression that is subsequently used to invoke the square operations: Function<Integer, Integer> memoizationFunction = Memoizer.memoize(squareFunction); System.out.println(memoizationFunction.apply(2)); System.out.println(memoizationFunction.apply(2)); System.out.println(memoizationFunction.apply(2)); The output of this sequence follows where the square operation is performed only once: In function 4 4 4 We can easily use the Memoizer class for a different function as shown here: Function<Double, Double> memoizationFunction2 = Memoizer.memoize(x -> x * x); System.out.println(memoizationFunction2.apply(4.0)); This will square the number as expected. Functions that are recursive present additional problems. Eliminating dependencies between functions When dependencies between functions are eliminated, then more flexibility in the order of execution is possible. Consider these Function and BiFunction declarations, which define simple expressions for computing hourly, salaried, and sales type pay, respectively: BiFunction<Integer, Double, Double> computeHourly = (hours, rate) -> hours * rate; Function<Double, Double> computeSalary = rate -> rate * 40.0; BiFunction<Double, Double, Double> computeSales = (rate, commission) -> rate * 40.0 + commission; These functions can be executed, and their results are assigned to variables as shown here: double hourlyPay = computeHourly.apply(35, 12.75); double salaryPay = computeSalary.apply(25.35); double salesPay = computeSales.apply(8.75, 2500.0); These are pure functions as they do not use external values to perform their computations. In the following code sequence, the sum of all three pays are totaled and displayed: System.out.println(computeHourly.apply(35, 12.75) + computeSalary.apply(25.35) + computeSales.apply(8.75, 2500.0)); We can easily reorder their execution sequence or even execute them concurrently, and the results will be the same. There are no dependencies between the functions that restrict them to a specific execution ordering. Supporting lazy evaluation Continuing with this example, let's add an additional sequence, which computes the total pay based on the type of employee. The variable, hourly, is set to true if we want to know the total of the hourly employee pay type. It will be set to false if we are interested in salary and sales-type employees: double total = 0.0; boolean hourly = ...; if(hourly) { total = hourlyPay; } else { total = salaryPay + salesPay; } System.out.println(total); When this code sequence is executed with an hourly value of false, there is no need to execute the computeHourly function since it is not used. The runtime system could conceivably choose not to execute any of the lambda expressions until it knows which one is actually used. While all three functions are actually executed in this example, it illustrates the potential for lazy evaluation. Functions are not executed until needed. Referential transparency Referential transparency is the idea that a given expression is made up of subexpressions. The value of the subexpression is important. We are not concerned about how it is written or other details. We can replace the subexpression with its value and be perfectly happy. With regards to pure functions, they are said to be referentially transparent since they have same effect. In the next declaration, we declare a pure function called pureFunction: Function<Double,Double> pureFunction = t -> 3*t; It supports referential transparency. Consider if we declare a variable as shown here: int num = 5; Later, in a method we can assign a different value to the variable: num = 6; If we define a lambda expression that uses this variable, the function is no longer pure: Function<Double,Double> impureFunction = t -> 3*t+num; The function no longer supports referential transparency. Closure in Java The use of external variables in a lambda expression raises several interesting questions. One of these involves the concept of closures. A closure is a function that uses the context within which it was defined. By context, we mean the variables within its scope. This sometimes is referred to as variable capture. We will use a class called ClosureExample to illustrate closures in Java. The class possesses a getStringOperation method that returns a Function lambda expression. This expression takes a string argument and returns an augmented version of it. The argument is converted to lowercase, and then its length is appended to it twice. In the process, both an instance variable and a local variable are used. In the implementation that follows, the instance variable and two local variables are used. One local variable is a member of the getStringOperation method and the second one is a member of the lambda expression. They are used to hold the length of the target string and for a separator string: public class ClosureExample { int instanceLength; public Function<String,String> getStringOperation() { final String seperator = ":"; return target -> { int localLength = target.length(); instanceLength = target.length(); return target.toLowerCase() + seperator + instanceLength + seperator + localLength; }; } } The lambda expression is created and used as shown here: ClosureExample ce = new ClosureExample(); final Function<String,String> function = ce.getStringOperation(); System.out.println(function.apply("Closure")); Its output follows: closure:7:7 Variables used by the lambda expression are restricted in their use. Local variables or parameters cannot be redefined or modified. These variables need to be effectively final. That is, they must be declared as final or not be modified. If the local variable and separator, had not been declared as final, the program would still be executed properly. However, if we tried to modify the variable later, then the following syntax error would be generated, indicating such variable was not permitted within a lambda expression: local variables referenced from a lambda expression must be final or effectively final If we add the following statements to the previous example and remove the final keyword, we will get the same syntax error message: function = String::toLowerCase; Consumer<String> consumer = s -> System.out.println(function.apply(s)); This is because the function variable is used in the Consumer lambda expression. It also needs to be effectively final, but we tried to assign a second value to it, the method reference for the toLowerCase method. Closure refers to functions that enclose variable external to the function. This permits the function to be passed around and used in different contexts. Currying Some functions can have multiple arguments. It is possible to evaluate these arguments one-by-one. This process is called currying and normally involves creating new functions, which have one fewer arguments than the previous one. The advantage of this process is the ability to subdivide the execution sequence and work with intermediate results. This means that it can be used in a more flexible manner. Consider a simple function such as: f(x,y) = x + y The evaluation of f(2,3) will produce a 5. We could use the following, where the 2 is "hardcoded": f(2,y) = 2 + y If we define: g(y) = 2 + y Then the following are equivalent: f(2,y) = g(y) = 2 + y Substituting 3 for y we get: f(2,3) = g(3) = 2 + 3 = 5 This is the process of currying. An intermediate function, g(y), was introduced which we can pass around. Let's see, how something similar to this can be done in Java 8. Start with a BiFunction method designed for concatenation of strings. A BiFunction method takes two parameters and returns a single value: BiFunction<String, String, String> biFunctionConcat = (a, b) -> a + b; The use of the function is demonstrated with the following statement: System.out.println(biFunctionConcat.apply("Cat", "Dog")); The output will be the CatDog string. Next, let's define a reference variable called curryConcat. This variable is a Function interface variable. This interface is based on two data types. The first one is String and represents the value passed to the Function interface's accept method. The second data type represents the accept method's return type. This return type is defined as a Function instance that is passed a string and returns a string. In other words, the curryConcat function is passed a string and returns an instance of a function that is passed and returns a string. Function<String, Function<String, String>> curryConcat; We then assign an appropriate lambda expression to the variable: curryConcat = (a) -> (b) -> biFunctionConcat.apply(a, b); This may seem to be a bit confusing initially, so let's take it one piece at a time. First of all, the lambda expression needs to return a function. The lambda expression assigned to curryConcat follows where the ellipses represent the body of the function. The parameter, a, is passed to the body: (a) ->...; The actual body follows: (b) -> biFunctionConcat.apply(a, b); This is the lambda expression or function that is returned. This function takes two parameters, a and b. When this function is created, the a parameter will be known and specified. This function can be evaluated later when the value for b is specified. The function returned is an instance of a Function interface, which is passed two parameters and returns a single value. To illustrate this, define an intermediate variable to hold this returned function: Function<String,String> intermediateFunction; We can assign the result of executing the curryConcat lambda expression using it's apply method as shown here where a value of Cat is specified for the a parameter: intermediateFunction = curryConcat.apply("Cat"); The next two statements will display the returned function: System.out.println(intermediateFunction); System.out.println(curryConcat.apply("Cat")); The output will look something similar to the following: packt.Chapter2$$Lambda$3/798154996@5305068a packt.Chapter2$$Lambda$3/798154996@1f32e575 Note that these are the values representing this functions as returned by the implied toString method. They are both different, indicating that two different functions were returned and can be passed around. Now that we have confirmed a function has been returned, we can supply a value for the b parameter as shown here: System.out.println(intermediateFunction.apply("Dog")); The output will be CatDog. This illustrates how we can split a two parameter function into two distinct functions, which can be evaluated when desired. They can be used together as shown with these statements: System.out.println(curryConcat.apply("Cat").apply("Dog")); System.out.println(curryConcat.apply("Flying ").apply("Monkeys")); The output of these statements is as follows: CatDog Flying Monkeys We can define a similar operation for doubles as shown here: Function<Double, Function<Double, Double>> curryAdd = (a) -> (b) -> a * b; System.out.println(curryAdd.apply(3.0).apply(4.0)); This will display 12.0 as the returned value. Currying is a valuable approach useful when the arguments of a function need to be evaluated at different times. Summary In this article, we investigated the use of lambda expressions and how they support the functional style of programming in Java 8. When possible, we used examples to contrast the use of classes and methods against the use of functions. This frequently led to simpler and more maintainable functional implementations. We illustrated how lambda expressions support the functional concepts of high-order, first-class, and pure functions. Examples were used to help clarify the concept of referential transparency. The concepts of closure and currying are found in most functional programming languages. We provide examples of how they are supported in Java 8. Lambda expressions have a specific syntax, which we examined in more detail. Also, there are several variations of the function that can be used to support the expression in the form, which we illustrated. Lambda expressions are based on functional interfaces using type inference. It is important to understand how to create functional interfaces and to know what standard functional interfaces are available in Java 8. Resources for Article: Further resources on this subject: An Introduction to Mastering JavaScript Promises and Its Implementation in Angular.js[article] Finding Peace in REST[article] Introducing JAX-RS API [article]
Read more
  • 0
  • 0
  • 23805
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-creating-data-dictionary-using-chatgpt
Sagar Lad
04 Jun 2023
4 min read
Save for later

Creating Data Dictionary Using ChatGPT

Sagar Lad
04 Jun 2023
4 min read
This article highlights how ChatGPT can create data dictionaries within minutes, aiding data professionals in documenting data items. By leveraging ChatGPT's capabilities, professionals gain deeper insights, enhancing data management. A practical example demonstrates the efficiency and effectiveness of using ChatGPT in generating data dictionaries. What is a data dictionary? Data professionals, such as data engineers, data scientists, analysts, database administrators, and developers, face various data challenges, ranging from business requirement definition to data volume and speed management. To effectively tackle these difficulties, they require a comprehensive understanding of the data. Data dictionaries play a vital role in providing deeper insights into the data. A data dictionary serves as documentation for the data, encompassing names, definitions, and attributes of the database's data items. Its main purpose is to comprehend and describe the significance of data items in relation to the application, along with including data element metadata. Data dictionaries are indispensable in data projects as they contribute to success by offering valuable insights into the data. Benefits of creating a data dictionary: Conquer data discrepancies Facilitate data exploration and analysisMaintain data standards throughout the projectEstablish uniform and consistent standards for the projectEstablish data standards to control the gathered data and explain it across the project A typical data dictionary has below components:  ComponentDescriptionData ElementName of the data elementDescription Definition of the data elementData TypeType of data stored in the attribute (ex. text, number, date)LengthMaximum number of characters stored in the attributeFormatFormat  for the data (e.g. date/currency format) Valid ValuesList of allowed values for the data elementRelationshipsRelationships between different tables in the databaseSource Origin of the data (e.g. system, department)ConstraintsRules related to the use of the dataListing of data objectsNames and DefinitionsDetailed properties of data elementsData type, Size, nullability, optionality, indexesBusiness rulesSchema validation or Data Quality Image 1 : Sample Database Schema with Data Attribute Name, Data Type & Constraints     As demonstrated in the example above, each database has a basic set of data about the data dictionary, but this information is insufficient when working with a database that has numerous tables, each of which may have multiple columns.  Creating a practical data dictionary with ChatGPT Data and natural language processing can be used by ChatGPT to produce in-depth knowledge on any subject. As a result, ChatGPT may be used to build instructive data dictionaries for any dataset.       Image 2: ChatGPT to create Data Dictionary Let’s understand the step-by-step process to create a data dictionary using ChatGPT: Finding and copying the data               Let us ask ChatGPT for one of the public datasets to create a data dictionary:    List of Data Sources recommended by ChatGPT Now, I will download the csv file named Institutions.csv from the FDIC Bank Data API. Image 4: Downloaded CSV file for FDIC Bank DataLet’s use this data to create a data dictionary using ChatGPT. Prepare ChatGPT Let’s now prompt the GPT to create a raw data dictionary for the dataset that we picked above:Image 5: Output Data Dictionary Request Data Dictionary with additional information Additionally, we can request that ChatGPT add new columns and other pertinent data to the output of the data dictionary. For instance, in the sample below, I've asked ChatGPT to add a new column called Active Loan and to provide descriptions to the columns based on its knowledge of banking. Output Data Dictionary from ChatGPT with additional columns and informationWe can now see that the data dictionary is updated which can be shared within the organization.ConclusionIn conclusion, leveraging ChatGPT's capabilities expedites the creation of data dictionaries, enhancing data management for professionals. Its efficiency and insights empower successful data projects, making ChatGPT a valuable tool in the data professional's toolkit. Author BioSagar Lad is a Cloud Data Solution Architect with a leading organization and has deep expertise in designing and building Enterprise-grade Intelligent Azure Data and Analytics Solutions. He is a published author, content writer, Microsoft Certified Trainer, and C# Corner MVP. Link - Medium, Amazon, LinkedIn.
Read more
  • 0
  • 0
  • 23778

article-image-generic-section-single-page-based-website
Savia Lobo
17 May 2018
14 min read
Save for later

How to create a generic reusable section for a single page based website

Savia Lobo
17 May 2018
14 min read
There are countless variations when it comes to different sections that can be incorporated into the design of a single page website. In this tutorial, we will cover how to create a generic section that can be extended to multiple sections. This section provides the ability to display any information your website needs. Single page sections are commonly used to display the following data to the user: Contact form (will be implemented in the next chapter). About us: This can be as simple as a couple of paragraphs talking about the company/individual or more complex with images, even showing the team and their roles. Projects/work: Any work you or the company has done and would like to showcase. They are usually linked to external pages or pop up boxes containing more information about the project. Useful company info such as opening times. These are just some of the many uses for sections in a single page website. A good rule of thumb is that if it can be a page on another website it can most likely be adapted into sections on a single page website. Also, depending on the amount of information a single section has, it could potentially be split into multiple sections. This article is an excerpt taken from the book,' Responsive Web Design by Example', written by Frahaan Hussain. Single page section examples Let's go through some examples of the sections mentioned above. Example 1: Contact form As can be seen, by the contact form from Richman, the elements used are very similar to that of a contact page. A form is used with inputs for the various pieces of information required from the user along with a button for submission: Not all contact forms will have the same fields. Put what you need, it may be more or less, there is no right or wrong answer. Also at the bottom of the section is the company's logo along with some written contact information, which is also very common. Some websites also display a map usually using the Google Maps API; these mainly have a physical presence such as a store. Website link—http://richman-kcm.com/ Example 2: About us This is an excellent example of an about us page that uses the following elements to convey the information: Images: Display the individual's face. Creates a very personal touch to the otherwise digital website. Title: Used to display the individual's name. This can also be an image if you want a fancier title. Simple text: Talks about who the person is and what they do. Icons: Linking to the individual's social media accounts. Website link—http://designedbyfew.com/ Example 3: Projects/work This website shows its work off very elegantly and cleanly using images and little text: It also provides a carousel-like slider to display the work, which is extremely useful for displaying the content bigger without displaying all of it at once and it allows a lot of content for a small section to be used. Website link: http://peeltheorange.com/#recent-work Example 4: Opening times This website uses a background image similar to the introduction section created in the previous chapter and an additional image on top to display the opening times. This can also be achieved using a mixture of text and CSS styling for various facets such as the border. Website link—http://www.mumbaigate.co.uk/ Implementing generic reusable single page section We will now create a generic section that can easily be modified and reused to our single page portfolio website. But we still need some sort of layout/design in mind before we implement the section, let's go with an Our Team style section. What will the Our Team section contain? The Our Team section will be a bit simpler, but it can easily be modified to accommodate the animations and styles displayed on the previously mentioned websites. It will be similar to the following example: Website link—http://demo.themeum.com/html/oxygen/ The preceding example consists of the following elements: Heading Intro text (Lorem Ipsum in this case) Images displaying each member of the team Team member's name Their role Text informing the viewer a little bit about them Social links We will also create our section using a similar layout. We are now finally going to use the column system to its full potential to provide a responsive experience using breakpoints. Creating the Our Team section container First let's implement a simple container, with the title and section introduction text, without any extra elements such as an image. We will then use this to link to our navigation bar. Add the following code to the jumbotron div: Let's go over what the preceding code is doing: Line 9 creates a container that is fluid, allowing it to span the browser's width fully. This can be changed to a regular container if you like. The id will be used very soon to link to the navigation bar. Line 10 creates a row in which our text elements will be stored. Line 11 creates a div that spans all the 12 columns on all screen sizes and centers the text inside of it. Line 12 creates a simple header for the Team section. Line 14 to Line 16 adds introduction text. I have put the first two sentences of "Lorem Ipsum..." inside of it, but you can put anything you like. All of this produces the following result: Anchoring the Team section to the navigation bar We will now link the navigation bar to the Team section. This will allow the user to navigate to the Team section without having to scroll up or down. At the moment, there is no need to scroll up, but when more content is added this can become a problem as a single page website can become quite long. Fortunately, we have already done the heavy lifting with the navigation bar through HTML and JavaScript, phew! First, let's change the name of the second button in the navigation bar to Team. Update the navigation bar like so: The navigation bar will now look as follows: Fantastic, our navigation bar is looking more like what you would see on a real website. Now let's change href to the same ID as the Team section, which was #TeamSection like so: Now when we click on any of the navigation buttons we get no JavaScript errors like we would have in the previous chapter. Also, it automatically scrolls to each section without any extra JavaScript code. Adding team pictures Now let's use images to showcase the team members. I will use the image from the following link for our employees, but in a real website you would obviously use different images: http://res.cloudinary.com/dmliyxggm/image/upload/v1511699813/John_vepwoz.png I have modified the image so all the background is removed and the image is trimmed, so it looks as follows: Up until now, all images that we have used have been stored on other websites such as CDN's, this is great, but the need may arise when the use of a custom image like the previous one is needed. We can either store it on a CDN, which is a very good approach, and I would recommend Cloudinary (http://cloudinary.com/), or we can store it locally, which we will do now. A CDN is a Content Delivery Network that has a sole purpose of delivering content such as images to other websites using the best and fastest servers available to a specific user. I would definitely recommend using one. Create a folder called Images and place the image using the following folder structure: Root CSS Images Team Thumbnails Thumbnails.png Index.php JS SNIPPETS This may seem like overkill, considering we only have one image, but as your website gets more complex you will store more images and having an intelligent folder structure/hierarchy will save an immense amount of time. Add the following code to the first row like so: The code we have added doesn't actually provide any visual changes as it is nothing but empty div classes. But these div classes will serve as structures for each team member and their respective content such as name and social links. We created a new row to group our new div classes. Inside each div we will represent each team member. The classes have been set up to be displayed like so: Extra small screens will only show a single team member on a single row Small and medium screens will show two team members on a single row Large and extra large screens will show four team members on a single row The rows are rows in their literal sense and not the class row. Another way to look at them is as lines. The sizes/breakpoints can easily be changed using the information regarding the grid from this article What Is Bootstrap, Why Do We Use It? Now let's add the team's images, update the previous code like so: The preceding code produces the following result: As you can see, this is not the desired effect we were looking for. As there are no size restrictions on the image, it is displayed at its original size. Which, on some screens, will produce a result similar to the monstrosity you saw before; worry not, this can easily be fixed. Add the classes img-fluid and img-thumbnail to each one of the images like so: The classes we added are designed to provide the following styling: img-fluid: Provides a responsive image that is automatically restricted based on the number of columns and browser size. img-thumbnail: Is more of an optional class, but it is still very useful. It provides a light border around the images to make them pop. This produces the following result: As it can be seen, this is significantly better than our previous result. Depending on the browser/screen size, the positioning will slightly change based on the column breakpoints we specified. As usual, I recommend that you resize the browser to see the different layouts. These images are almost complete; they look fine on most screen sizes, but they aren't actually centered within their respective div. This is evident in larger screen sizes, as can be seen here: It isn't very noticeable, but the problem is there, it can be seen to the right of the last image. You probably could get away without fixing this, but when creating anything, from a website to a game, or even a table, the smallest details are what separate the good websites from the amazing websites. This is a simple idea called the aggregation of marginal gains. Fortunately for us, like many times before, Bootstrap offers functionality to resolve our little problem. Simply add the text-center class, to the row within the div of the images like so:   This now produces the following result: There is one more slight problem that is only noticeable on smaller screens when the images/member containers are stacked on top of each other. The following result is produced: The problem might not jump out at first glance, but look closely at the gaps between the images that are stacked, or I should say, to the lack of a gap. This isn't the end of the world, but again the small details make an immense difference to the look of a website. This can be easily fixed by adding padding to each team member div. First, add a class of teamMemberContainer to each team member div like so: Add the following CSS code to the index.css file to provide a more visible gap through the use of padding: This simple solution now produces the following result: If you want the gap to be bigger, simply increase the value and lower it to reduce the gap. Team member info text The previous section covered quite a lot if you're not 100% on what we did just go back and take a second look. This section will thankfully be very simple as it will incorporate techniques and features we have already covered, to add the following information to each team member: Name Job title Member info text Plus anything else you need Update each team member container with the following code: Let's go over the new code line by line: Line 24 adds a simple header that is intended to display the team member's name. I have chosen an h4 tag, but you can use something bigger or smaller if you like. Line 26 adds the team member's job title, I have used a paragraph element with the Bootstrap class text-muted, which lightens the text color. If you would like more information regarding text styling within Bootstrap, feel free to check out the following link. Line 28 adds a simple paragraph with no extra styling to display some information about the team member. Bootstrap text styling link—https://v4-alpha.getbootstrap.com/utilities/colors/ The code that we just added will produce the following result: As usual, resize your browser to simulate different screen sizes. I use Chrome as my main browser, but Safari has an awesome feature baked right in that allows you to see how your website will run on different browsers/devices, this link will help you use this feature—https://www.tekrevue.com/tip/safari-responsive-design-mode/ Most browsers have a plethora of plugins to aid in this process, but not only does Safari have it built in, it works really well. It all looks fantastic, but again I will nitpick at the gaps. The image is right on top of the team member name text; a small gap would really help improve the visual fidelity. Add a class of teamMemberImage to each image tag as it is demonstrated here: Now add the following code to the index.css file, which will apply a margin of 10px below the image, hence moving all the content down:  Change the margin to suit your needs. This very simple code will produce the following similar yet subtly different and more visually appealing result: Team member social links We have almost completed the Team section, only the social links remain for each team member. I will be using simple images for the social buttons from the following link: https://simplesharebuttons.com/html-share-buttons/ I will also only be adding three social icons, but feel free to add as many or as few as you need. Add the following code to the button of each team member container: Let's go over each new line of code: Line 30 creates a div to store all the social buttons for each team member Line 31 creates a link to Facebook (add your social link in the href) Line 32 adds an image to show the Facebook social link Line 35 creates a link to Google+ (add your social link in the href) Line 36 adds an image to show the Google+ social link Line 39 creates a link to Twitter (add your social link in the href) Line 40 adds an image to show the Twitter social link We have added a class that needs to be implemented, but let's first run our website to see the result without any styling: It looks OK, but the social icons are a bit big, especially if we were to have more icons. Add the following CSS styling to the index.css file: This piece of code simply restricts the social icons size to 50px. Only setting the width causes the height to be automatically calculated, this ensures that any changes to the image that involves a ratio change won't mess up the look of the icons. This now produces the following result: Feel free to change the width to suit your desires. With the social buttons implemented, we are done. If you've enjoyed this tutorial, check out Responsive Web Design by Example, to create a cool blog page, beautiful portfolio site, or professional business site to make them all totally responsive. 5 things to consider when developing an eCommerce website What UX designers can teach Machine Learning Engineers? To start with: Model Interpretability
Read more
  • 0
  • 0
  • 23743

article-image-using-rest-api-unity-part-1-what-rest-and-basic-queries
Denny and
24 Jun 2015
6 min read
Save for later

Using a REST API with Unity Part 1

Denny and
24 Jun 2015
6 min read
Introduction While developing a game, there a number of reasons why you would want to connect to a server. Downloading new assets, such as models, or collecting data from an external source is one reason. Downloading bundle assets can be done through your own server, which allows your game to connect to a server and download the most recent versions of bundle assets. Suppose your game also allowed users to see if that item was available at Amazon, and for what price? If you had the sku number available, you could connect to Amazon's API, and check the price and availability of that item. The most common way to connect to external API's these days, is through a RESTful API. What is a REST API A RESTful api is a common approach to creating scalable web services. It provides users with endpoints to collect and create data using the same HTTP calls to collect web pages (GET, POST, PUT, DELETE). For example, a url like www.fake.com/users could return a JSON of User data. Of course, there is often more security involved with these calls, but this is a good starting point. Once you begin understanding REST API's, it becomes very second nature to query them. Before doing anything in code, you can try a query! Go to the browser and go to the url: http://jsonplaceholder.typicode.com/posts. You should be returned a JSON of some post data. You can see REST endpoints in action already. Your browser posted a GET request to the /posts endpoint, which returns all the posts. What if we want just a specific post? The standard way to do this is to add the id of the post next. Like this: http://jsonplaceholder.typicode.com/posts/1. You should get just a single post this time. Great! Often when building Unity scripts to connect to a REST endpoint, we'll frequently use this site to test on, before we move to the actual REST endpoints I want! Setting up your own server Setting up our own server is a bit out of the scope of this article. In previous projects, we've used a framework like Sails JS to create a Node Server, with REST endpoints. Parsing JSON in Unity Basic REST One of the worst parts of querying REST data is the parsing in Unity. Compared to parsing JSON on the web, Unity's parsing can feel tricky. The primary tool we use to make life a little easier is called SimpleJSON. It allows us to create JSON objects in C#, which we can use to build or read JSON data, and then manipulate them to our need. We won't be going into detail on how to use SimpleJSON, as much as just using the data retrieved from it. For further reading, we recommend looking at their documentation. Just to note though, SimpleJSON does not allow for parsing of GameObjects and such in Unity, instead it deals with only more primitive attributes like strings and floats. For example, let's assume we wanted to upload a list of products to a server from our Unity project, without the game running (in editor). Assuming we collected the data from our game and its currently residing in a JSON file, let's see the code on how we can upload this data to the server from Unity. string productlist = File.ReadAllText(Application.dataPath + "/Resources/AssetBundles/" + "AssetBundleInfo.json"); UploadProductListJSON(productList); static void UploadProductListJSON(string data) { Debug.Log (data); WWWForm form = new WWWForm(); form.AddField("productlist", data); WWW www = new WWW("localhost:1337/product/addList", form); } So we pass the collected data to a function that will create a new form, add the data to that form and then use the WWW variable to upload our form to the server. This will use the POST request to add new data. We normally don't want to create a different end point to add data, such as /addList. We could have added data one at a time, and used the standard REST endpoint (/product). This would likely be the cleaner solution, but for the sake of simplicity, we've added an endpoint that accepts a list of data. Building REST Factories for In Game REST Calls Rather than having random scripts containing API calls, we recommend following the standard web procedure and building REST factories. Scripts with the sole purpose of querying rest endpoints. When contacting a server from in game, the standard approach is to use a coroutine, as to not lock your game on the thread. Let's take a look at the standard DB factory we use. private string results; public String Results { get { return results; } } public WWW GET(string url, System.Action onComplete ) { WWW www = new WWW (url); StartCoroutine (WaitForRequest (www, onComplete)); return www; } public WWW POST(string url, Dictionary<string,string> post, System.Action onComplete) { WWWForm form = new WWWForm(); foreach(KeyValuePair<string,string> post_arg in post) { form.AddField(post_arg.Key, post_arg.Value); } WWW www = new WWW(url, form); StartCoroutine(WaitForRequest(www, onComplete)); return www; } private IEnumerator WaitForRequest(WWW www, System.Action onComplete) { yield return www; // check for errors if (www.error == null) { results = www.text; onComplete(); } else { Debug.Log (www.error); } } The url data here would be something like our example above: http://jsonplaceholder.typicode.com/posts. The System.Action OnComplete is a callback to be called once the action is complete. This will normally be some method that requires the downloaded data. In both our GET and POST methods, we will connect to a passed URL, and then pass our www objects to a co-routine. This will allow our game to continue while the queries are being resolved in the WaitForRequest method. This method will either collect the result, and call any callbacks, or it will log the error for us. Conclusion This just touches the basics of building a game that allows connecting and usage of REST endpoints. In later editions, we can talk about building a thorough, modular system to connect to REST endpoints, extracting meaningful data from your queries using simple JSON, user authentication, and how to build a manager system to handle multiple REST calls. About the Authors Denny is a Mobile Application Developer at Canadian Tire Development Operations. While working, Denny regularly uses Unity to create in-store experiences, but also works on other technologies like Famous, Phaser.IO, LibGDX, and CreateJS when creating game-like apps. He also enjoys making non-game mobile apps, but who cares about that, am I right? Travis is a Software Engineer, living in the bitter region of Winnipeg, Canada. His work and hobbies include Game Development with Unity or Phaser.IO, as well as Mobile App Development. He can enjoy a good video game or two, but only if he knows he'll win!
Read more
  • 0
  • 1
  • 23739

article-image-malicious-code-in-npm-event-stream-package-targets-a-bitcoin-wallet-and-causes-8-million-downloads-in-two-months
Savia Lobo
28 Nov 2018
3 min read
Save for later

Malicious code in npm ‘event-stream' package targets a bitcoin wallet and causes 8 million downloads in two months

Savia Lobo
28 Nov 2018
3 min read
Last week Ayrton Sparling, a Computer Science major at CSUF, California disclosed that the popular npm package, event-stream, contains a malicious package named flatmap-stream. He disclosed the issue via the GitHub issue on the EventStream’s repository. The event-stream npm package was originally created and maintained by Dominic Tarr. However, this popular package has not been updated for a long time now. According to Thomas Hunter’s post on Medium, “Ownership of event-stream, was transferred by the original author to a malicious user, right9ctrl.  The malicious user was able to gain the trust of the original author by making a series of meaningful contributions to the package.” The malicious owner then added a malicious library named flatmap-stream to the events-stream package as a dependency. This led to a download and invocation of the event-stream package (using the malicious 3.3.6 version) by every user. The malicious library download added up to nearly 8 million downloads since it was included in September 2018. The malicious package represents a highly targeted attack and affects an open source app called bitpay/copay. Copay is a secure bitcoin wallet platform for both desktop and mobile devices. “We know the malicious package specifically targets that application because the obfuscated code reads the description field from a project’s package.json file, then uses that description to decode an AES256 encrypted payload”, said Thomas in his post. Post this breakout, many users from Twitter and GitHub have positively supported Dominic. In a statement on the event-stream issue, Dominic stated, “I've shared publish rights with other people before. Of course, If I had realized they had a malicious intent I wouldn't have, but at the time it looked like someone who was actually trying to help me”. https://twitter.com/dominictarr/status/1067186943304159233 As a support to Dominic, André Staltz, an open source hacker, tweeted, https://twitter.com/andrestaltz/status/1067157915398746114 Users affected by this malicious code are advised to eliminate this package from their application by reverting back to version 3.3.4 of event-stream. If the user application deals with Bitcoin, they should inspect its activity in the last 3 months to see if any mined or transferred bitcoins did not make it into their wallet. However, if the application does not deal with bitcoin but is especially sensitive, an inspection of its activity in the last 3 months for any suspicious activity is recommended. This is to analyze the notably data sent on the network to unintended destinations. To know more about this in detail, visit Eventstream’s repository. A new data breach on Facebook due to malicious browser extensions allowed almost 81,000 users’ private data up for sale, reports BBC News Wireshark for analyzing issues and malicious emails in POP, IMAP, and SMTP [Tutorial] Machine learning based Email-sec-360°surpasses 60 antivirus engines in detecting malicious emails
Read more
  • 0
  • 0
  • 23736
article-image-26-new-java-9-enhancements-you-will-love
Aarthi Kumaraswamy
09 Apr 2018
11 min read
Save for later

26 new Java 9 enhancements you will love

Aarthi Kumaraswamy
09 Apr 2018
11 min read
Java 9 represents a major release and consists of a large number of internal changes to the Java platform. Collectively, these internal changes represent a tremendous set of new possibilities for Java developers, some stemming from developer requests, others from Oracle-inspired enhancements. In this post, we will review 26 of the most important changes. Each change is related to a JDK Enhancement Proposal (JEP). JEPs are indexed and housed at openjdk.java.net/jeps/0. You can visit this site for additional information on each JEP. [box type="note" align="" class="" width=""]The JEP program is part of Oracle's support for open source, open innovation, and open standards. While other open source Java projects can be found, OpenJDK is the only one supported by Oracle. [/box] These changes have several impressive implications, including: Heap space efficiencies Memory allocation Compilation process improvements Type testing Annotations Automated runtime compiler tests Improved garbage collection 26 Java 9 enhancements you should know Improved Contended Locking [JEP 143] The general goal of JEP 143 was to increase the overall performance of how the JVM manages contention over locked Java object monitors. The improvements to contended locking were all internal to the JVM and do not require any developer actions to benefit from them. The overall improvement goals were related to faster operations. These include faster monitor enter, faster monitor exit, and faster notifications. 2. Segmented code cache [JEP 197] The segmented code cache JEP (197) upgrade was completed and results in faster, more efficient execution time. At the core of this change was the segmentation of the code cache into three distinct segments--non-method, profiled, and non-profiled code. 3. Smart Java compilation, phase two [JEP 199] The JDK Enhancement Proposal 199 is aimed at improving the code compilation process. All Java developers will be familiar with the javac tool for compiling source code to bytecode, which is used by the JVM to run Java programs. Smart Java Compilation, also referred to as Smart Javac and sjavac, adds a smart wrapper around the javac process. Perhaps the core improvement sjavac adds is that only the necessary code is recompiled. [box type="shadow" align="" class="" width=""]Check out this tutorial to know how you can recognize patterns with neural networks in Java.[/box] 4. Resolving Lint and Doclint warnings [JEP 212] Both Lint and Doclint report errors and warnings during the compile process. Resolution of these warnings was the focus of JEP 212. When using core libraries, there should not be any warnings. This mindset led to JEP 212, which has been resolved and implemented in Java 9. 5. Tiered attribution for javac [JEP 215] JEP 215 represents an impressive undertaking to streamline javac's type checking schema. In Java 8, type checking of poly expressions is handled by a speculative attribution tool. The goal with JEP 215 was to change the type checking schema to create faster results. The new approach, released with Java 9, uses a tiered attribution tool. This tool implements a tiered approach for type checking argument expressions for all method calls. Permissions are also made for method overriding. 6. Annotations pipeline 2.0 [JEP 217] Java 8 related changes impacted Java annotations but did not usher in a change to how javac processed them. There were some hardcoded solutions that allowed javac to handle the new annotations, but they were not efficient. Moreover, this type of coding (hardcoding workarounds) is difficult to maintain. So, JEP 217 focused on refactoring the javac annotation pipeline. This refactoring was all internal to javac, so it should not be evident to developers. 7. New version-string scheme [JEP 223] Prior to Java 9, the release numbers did not follow industry standard versioning--semantic versioning. Oracle has embraced semantic versioning for Java 9 and beyond. For Java, a major-minor-security schema will be used for the first three elements of Java version numbers: Major: A major release consisting of a significant new set of features Minor: Revisions and bug fixes that are backward compatible Security: Fixes deemed critical to improving security 8. Generating run-time compiler tests automatically [JEP 233] The purpose of JEP 233 was to create a tool that could automate the runtime compiler tests. The tool that was created starts by generating a random set of Java source code and/or byte code. The generated code will have three key characteristics: Be syntactically correct Be semantically correct Use a random seed that permits reusing the same randomly-generated code 9. Testing class-file attributes generated by Javac [JEP 235] Prior to Java 9, there was no method of testing a class-file's attributes. Running a class and testing the code for anticipated or expected results was the most commonly used method of testing javac generated class-files. This technique falls short of testing to validate the file's attributes. The lack of, or insufficient, capability to create tests for class-file attributes was the impetus behind JEP 235. The goal is to ensure javac creates a class-file's attributes completely and correctly. 10. Storing interned strings in CDS archives [JEP 250] CDS archives now allocate specific space on the heap for strings: The string space is mapped using a shared-string table, hash tables, and deduplication. 11. Preparing JavaFX UI controls and CSS APIs for modularization [JEP 253] Prior to Java 9, JavaFX controls as well as CSS functionality were only available to developers by interfacing with internal APIs. Java 9's modularization has made the internal APIs inaccessible. Therefore, JEP 253 was created to define public, instead of internal, APIs. This was a larger undertaking than it might seem. Here are a few actions that were taken as part of this JEP: Moving javaFX control skins from the internal to public API (javafx.scene.skin) Ensuring API consistencies Generation of a thorough javadoc 12. Compact strings [JEP 254] The string data type is an important part of nearly every Java app. While JEP 254's aim was to make strings more space-efficient, it was approached with caution so that existing performance and compatibilities would not be negatively impacted. Starting with Java 9, strings are now internally represented using a byte array along with a flag field for encoding references. 13. Merging selected Xerces 2.11.0 updates into JAXP [JEP 255] Xerces is a library used for parsing XML in Java. It was updated to 2.11.0 in late 2010, so JEP 255's aim was to update JAXP to incorporate changes in Xerces 2.11.0. 14. Updating JavaFX/Media to a newer version of GStreamer [JEP 257] The purpose of JEP 257 was to ensure JavaFX/Media was updated to include the latest release of GStreamer for stability, performance, and security assurances. GStreamer is a multimedia processing framework that can be used to build systems that take in media from several different formats and, after processing, export them in selected formats. 15. HarfBuzz Font-Layout Engine [JEP 258] Prior to Java 9, the layout engine used to handle font complexities; specifically, fonts that have rendering behaviors beyond what the common Latin fonts have. Java used the uniform client interface, also referred to as ICU, as the defacto text rendering tool. The ICU layout engine has been depreciated and, in Java 9, has been replaced with the HarfBuzz font layout engine. HarfBuzz is an OpenType text rendering engine. This type of layout engine has the characteristic of providing script-aware code to help ensure text is laid out as desired. 16. HiDPI graphics on Windows and Linux [JEP 263] JEP 263 was focused on ensuring the crispness of on-screen components, relative to the pixel density of the display. The following terms are relevant to this JEP and are provided along with the below listed descriptive information: DPI-aware application: An application that is able to detect and scale images for the display's specific pixel density DPI-unaware application: An application that makes no attempt to detect and scale images for the display's specific pixel density HiDPI graphics: High dots-per-inch graphics Retina display: This term was created by Apple to refer to displays with a pixel density of at least 300 pixels per inch Prior to Java 9, automatic scaling and sizing were already implemented in Java for the Mac OS X operating system. This capability was added in Java 9 for Windows and Linux operating systems. 17. Marlin graphics renderer [JEP 265] JEP 265 replaced the Pisces graphics rasterizer with the Marlin graphics renderer in the Java 2D API. This API is used to draw 2D graphics and animations. The goal was to replace Pisces with a rasterizer/renderer that was much more efficient and without any quality loss. This goal was realized in Java 9. An intended collateral benefit was to include a developer-accessible API. Previously, the means of interfacing with the AWT and Java 2D was internal. 18. Unicode 8.0.0 [JEP 267] Unicode 8.0.0 was released on June 17, 2015. JEP 267 focused on updating the relevant APIs to support Unicode 8.0.0. In order to fully comply with the new Unicode standard, several Java classes were updated. The following listed classes were updated for Java 9 to comply with the new Unicode standard: java.awt.font.NumericShaper java.lang.Character java.lang.String java.text.Bidi java.text.BreakIterator java.text.Normalizer 19. Reserved stack areas for critical sections [JEP 270] The goal of JEP 270 was to mitigate problems stemming from stack overflows during the execution of critical sections. This mitigation took the form of reserving additional thread stack space. [box type="shadow" align="" class="" width=""]Are you looking out for running parallel data operations using Java streams, check out this post for more details.[/box] 20. Dynamic linking of language-defined object models [JEP 276] Java interoperability was enhanced with JEP 276. The necessary JDK changes were made to permit runtime linkers from multiple languages to coexist in a single JVM instance. This change applies to high-level operations, as you would expect. An example of a relevant high-level operation is the reading or writing of a property with elements such as accessors and mutators. The high-level operations apply to objects of unknown types. They can be invoked with INVOKEDYNAMIC instructions. Here is an example of calling an object's property when the object's type is unknown at compile time:   INVOKEDYNAMIC "dyn:getProp:age" 21. Additional tests for humongous objects in G1 [JEP 278] One of the long-favored features of the Java platform is the behind the scenes garbage collection. JEP 278's focus was to create additional WhiteBox tests for humongous objects as a feature of the G1 garbage collector. 22. Improving test-failure troubleshooting [JEP 279] For developers that do a lot of testing, JEP 279 is worth reading about. Additional functionality has been added in Java 9 to automatically collect information to support troubleshooting test failures as well as timeouts. Collecting readily available diagnostic information during tests stands to provide developers and engineers with greater fidelity in their logs and other output. 23. Optimizing string concatenation [JEP 280] JEP 280 is an interesting enhancement for the Java platform. Prior to Java 9, string concatenation was translated by javac into StringBuilder : : append chains. This was a sub-optimal translation methodology often requiring StringBuilder presizing. The enhancement changed the string concatenation bytecode sequence, generated by javac, so that it uses INVOKEDYNAMIC calls. The purpose of the enhancement was to increase optimization and to support future optimizations without the need to reformat the javac's bytecode. 24. HotSpot C++ unit-test framework [JEP 281] HotSpot is the name of the JVM. This Java enhancement was intended to support the development of C++ unit tests for the JVM. Here is a partial, non-prioritized, list of goals for this enhancement: Command-line testing Create appropriate documentation Debug compile targets Framework elasticity IDE support Individual and isolated unit testing Individualized test results Integrate with existing infrastructure Internal test support Positive and negative testing Short execution time testing Support all JDK 9 build platforms Test compile targets Test exclusion Test grouping Testing that requires the JVM to be initialized Tests co-located with source code Tests for platform-dependent code Write and execute unit testing (for classes and methods) This enhancement is evidence of the increasing extensibility. 25. Enabling GTK 3 on Linux [JEP 283] GTK+, formally known as the GIMP toolbox, is a cross-platform tool used for creating Graphical User Interfaces (GUI). The tool consists of widgets accessible through its API. JEP 283's focus was to ensure GTK 2 and GTK 3 were supported on Linux when developing Java applications with graphical components. The implementation supports Java apps that employ JavaFX, AWT, and Swing. 26. New HotSpot build system [JEP 284] The Java platform used, prior to Java 9, was a build system riddled with duplicate code, redundancies, and other inefficiencies. The build system has been reworked for Java 9 based on the build-infra framework. In this context, infra is short for infrastructure. The overarching goal for JEP 284 was to upgrade the build system to one that was simplified. Specific goals included: Leverage existing build system Maintainable code Minimize duplicate code Simplification Support future enhancements Summary We explored some impressive new features of the Java platform, with a specific focus on javac, JDK libraries, and various test suites. Memory management improvements, including heap space efficiencies, memory allocation, and improved garbage collection represent a powerful new set of Java platform enhancements. Changes regarding the compilation process resulting in greater efficiencies were part of this discussion We also covered important improvements, such as with the compilation process, type testing, annotations, and automated runtime compiler tests. You just enjoyed an excerpt from the book, Mastering Java 9 written by By Dr. Edward Lavieri and Peter Verhas.  
Read more
  • 0
  • 0
  • 23716

article-image-digital-and-mobile-forensics
Packt
14 Nov 2016
14 min read
Save for later

Digital and Mobile Forensics

Packt
14 Nov 2016
14 min read
In this article, Mattia Epifani and Pasquale Stirparo, co-authors of the book Learning iOS Forensics - Second Edition, would be talking mainly, if not solely, about computer forensics and computer crimes, such as when an attacker breaks into a computer network system and steals data. This would involve two types of offenses—unlawful/unauthorized access and data theft. As mobile phones became more popular, the new field of mobile forensics developed. (For more resources related to this topic, see here.) Nowadays, things have changed radically and they are still changing at quite a fast pace as technology evolves. Digital forensics, which includes all disciplines dealing with electronic evidence, is also being applied to common crimes, to those that, at least by definition, are not strictly IT crimes. Today, more than ever, we live in a society that is fully digitalized and people are equipped with all kinds of devices, which have different types of capabilities, but all of them process, store, and transmit information (mainly over the Internet). This means that forensic investigators have to be able to deal with all these devices. As defined at the first Digital Forensics Research Workshop (DFRWS) in 2001, digital forensics is: "The use of scientifically derived and proven methods toward the preservation, collection, validation, identification, analysis, interpretation, documentation, and presentation of digital evidence derived from digital sources for the purpose of facilitating or furthering the reconstruction of events found to be criminal, or helping to anticipate unauthorized actions shown to be disruptive to planned operations." As Casey asserted (Casey, 2011): "In this modern age, it is hard to imagine a crime that does not have a digital dimension." Criminals of all kinds use technology to facilitate their offenses, communicate with their peers, recruit other criminals, launder money, commit credit card fraud, gather information on their victims, and so on. This obviously creates new challenges for all the different actors involved, such as attorneys, judges, law enforcement agents, and forensic examiners. Among the cases solved in recent years, there were kidnappings where the kidnapper was caught—thanks to a request for ransom sent by e-mail from his mobile phone. There have been many cases of industrial espionage in which unfaithful employees were hiding projects in the memory cards of their smartphones, cases of drug dealing solved—thanks to the evidence found in the backup of mobile phones that was on computer, and many other such cases. Even the largest robberies of our time are now being conducted via computer networks. In this article, you will learn the following: Definition and principles of mobile forensics How to properly handle digital evidence Mobile forensics Mobile forensics is a field of study in digital forensics that focuses on mobile devices. Among the different digital forensics fields, mobile forensics is without doubt the fastest growing and evolving area of study, having an impact on many different situations from corporate to criminal investigations and intelligence gathering, which are on the rise. Moreover, the importance of mobile forensics is increasing exponentially due to the continuous fast growth of the mobile market. One of the most interesting peculiarities of mobile forensics is that mobile devices, particularly mobile phones, usually belong to a single individual, while this is not always the case with a computer that may be shared among employees of a company or members of a family. For this reason, the analysis of mobile phones gives access to plenty of personal information. Another important and interesting aspect that comes with mobile forensics, which is both challenging and frustrating at the same time for the analyst, is the multitude of different device models and the customized flavors of their operating systems available in the market. This makes it very difficult to have a single solution (either a tool or process) to address them all. Just think of all the applications people have installed on their smartphones: IM clients, web browsers, social network clients, password managers, navigation systems, and much more, other than the classic default ones, such as an address book, which can provide a lot more information than just the phone number for each contact that has been saved. Moreover, syncing such devices with a computer has become a very easy and smooth process, and all user activities, schedules, to-do lists, and everything else is stored inside a smartphone. Aren't these enough to profile a person and reconstruct all their recent activities, than building the network of contacts? Finally, in addition to a variety of smartphones and operating systems, such as Apple iOS, Google Android, Microsoft Windows Phone, and Blackberry OS, there is a massive number of so-called feature phones that use older mobile OS systems. Therefore, it's pretty clear that when talking about mobile/smartphone forensics, there is so much more than just printouts of phone calls. In fact, with a complete examination, we can retrieve SMSes/MMSes, pictures, videos, installed applications, e-mails, geolocation data, and so on—both present and deleted information. Digital evidence As mentioned earlier, on one hand the increasing involvement of mobile devices in digital forensics cases has brought a whole new series of challenges and complexities. However, on the other hand, this has also resulted in a much greater amount of evidence from criminals that it is now being used to reconstruct their activities with a more comprehensive level of detail. Moreover, while classical physical evidence may be destroyed, digital evidence, most of the time, leaves traces. Over the years, there have been several definitions of what digital evidence actually is, some of them focusing particularly on the evidentiary aspects of proof to be used in court, such as the one proposed by the Standard Working Group on Digital Evidence (SWGDE), stating that: "Digital evidence is any information of probative value that is either stored or transmitted in a digital form." The definition proposed by the International Organization of Computer Evidence (IOCE) states: "Digital evidence is information stored or transmitted in binary form that may be relied on in court." The definition given by E. Casey (Casey, 2000), refers to digital evidence as: "Physical objects that can establish that a crime has been committed, can provide a link between a crime and its victim, or can provide a link between a crime and its perpetrator." While all of these are correct, as previously said, all of these definitions focus mostly on proofs and tend to disregard data that is extremely useful for an investigation. For this reason, and for the purpose of this book, we will refer to the definition given by Carrier (Carrier, 2006), where digital evidence is defined as: "Digital data that supports or refutes a hypothesis about digital events or the state of digital data." This definition is a more general one, but better matches the current state of digital evidence and its value within the entire investigation process. Also from a standardization point of view, there have been, and still are, many attempts to define guidelines and best practices for digital forensics on how to handle digital evidence. Other than the several guidelines and special publications from NIST, there is a standard from ISO/IEC that was released in 2012, the ISO 27037 guidelines for identification, collection and/or acquisition, and preservation of digital evidence, which is not specific to mobile forensics, but is related to digital forensics in general, aiming to build a standard procedure for collecting and handling digital evidence, which will be legally recognized and accepted in court in different countries. This is a really important goal if you consider the lack of borders in the Internet era, particularly when it comes to digital crimes, where illicit actions can be perpetrated by attackers from anywhere in the world. Handling of mobile evidence In order to be useful not only in court but also during the entire investigation phase, digital evidence must be collected, preserved, and analyzed in a forensically sound manner. This means that each step, from the identification to the reporting, has to be carefully and strictly followed. Historically, we are used to referring to a methodology as forensically sound if, and only if, it would imply that the original source of evidence remains unmodified and unaltered. This was mostly true when talking about classical computer forensics, in scenarios where the forensic practitioner found the computer switched off or had to deal with external hard drives, although not completely true even in these situations. However, since the rise of live forensics, this concept has become more and more untrue. In fact, methods and tools for acquiring memory from live systems inevitably alter, even if just a little bit, the target system they are run on. The advent of mobile forensics stresses this concept even more, because mobile devices, and smartphones in particular, are networked devices that continuously exchange data through several communication protocols, such as GSM/CDMA, Wi-Fi, Bluetooth, and so on. Moreover, in order to acquire a mobile device, forensic practitioners need to have some degree of interaction with the device. Based on the type, a smartphone can need more or less interaction, altering in this way the original state of the device. All of this does not mean that preservation of the source evidence is useless, but that it is nearly impossible in the field of mobile devices. Therefore, it becomes a matter of extreme importance to thoroughly document every step taken during the collection, preservation, and acquisition phases. Using this approach, forensic practitioners will be able to demonstrate that they have been as unintrusive as possible. As Casey states (Casey, 2011): "One of the keys to forensic soundness is documentation. A solid case is built on supporting documentation that reports on where the evidence originated and how it was handled. From a forensic standpoint, the acquisition process should change the original evidence as little as possible and any changes should be documented and assessed in the context of the final analytical results." When in the presence of mobile devices to be collected, it is a good practice for the forensic practitioner to consider the following points: Take note of the current location where the device has been found. Report the device status (switched on or off, broken screen, and so on). Report date, time, and other information visible on the screen if the device is switched on, for example, by taking a picture of the screen. Look very carefully for the presence of memory cards. Although it is not the case with iOS devices, generally many mobile phones have a slot for an external memory card, where pictures, chat databases, and many other types of user data are usually stored. Look very carefully for the presence of cables related to the mobile phone that is being collected, especially if you don't have a full set of cables in your lab. Many mobile phones have their own cables to connect to the computer and to recharge the battery. Search for the original Subscriber Identity Module (SIM) package, because that is where the PIN and PIN unblocking key (PUK) codes are written. Take pictures of every item before collection. Modifications to mobile devices can happen not only because of interaction with the forensic practitioner, but also due to interaction with the network, voluntarily or not. In fact, digital evidence in mobile devices can be lost completely as they are susceptible to being overwritten by new data, for example, with the smartphone receiving an SMS while it is being collected, thus overwriting possible evidence previously stored in the same area of memory as the newly arrived SMS, or upon receiving a remote wiping command over a wireless network. Most of today's smartphones and iOS devices can be configured to be completely wiped remotely. From a real case: While searching inside the house of a person under investigation, law enforcement agents found and seized, among other things, computers and a smartphone. After cataloguing and documenting everything, they put all the material into boxes to bring them back to the laboratory. Once back in their laboratory, when acquiring the smart phone in order to proceed with the forensics analysis, they noticed that the smartphone was empty and it appeared to be brand new. The owner had wiped it remotely. Therefore, isolating the mobile device from all radio networks is a fundamental step in the process of preservation of evidence. There are several ways to achieve this, all with their own pros and cons, as follows: Airplane mode: Enabling Airplane mode on a device requires some sort of interaction, which may pose some risks of modification by the forensic practitioner. This is one of the best possible options since it implies that all wireless communication chips are switched off. In this case, it is always good to document the action taken with pictures and/or videos. Normally, this is possible only if the phone is not password-protected or the password is known. However, for devices with iOS 7 or higher, it is also possible to enable airplane mode by lifting the dock from the bottom, where there will be a button with the shape of a plane. This is possible only if the Access on Lock Screen option is enabled from Settings | Control Center. Faraday's bag: This item is a sort of envelope made of conducting material, which blocks out static electric fields and electromagnetic radiation completely isolating the device from communicating with external networks. It is based, as the name suggests, on Faraday's law. This is the most common solution, particularly useful when the device is being carried from the crime scene to the lab after seizure. However, the use of Faraday's bag will make the phone continuously search for a network, which will cause the battery to quickly drain. Unfortunately, it is also risky to plug the phone to a power cable outside that will go inside the bag, because this may act as antenna. Moreover, it is important to keep in mind that when you remove the phone from the bag (once arrived in the lab) it will again be exposed to the network. So, you would need either a shielded lab environment or a Faraday solution that would allow you to access the phone while it is still inside the shielded container, without the need for external power cables. Jamming: A jammer is used to prevent a wireless device from communicating by sending out radio waves along the same frequencies as that device. In our case, it would jam the GSM/UMTS/LTE frequencies that mobile phones use to connect with cellular base stations to send/receive data. Be aware that this practice may be considered illegal in some countries, since it will also interfere with any other mobile device in the range of the jammer, disrupting their communications too. Switching off the device: This is a very risky practice because it may activate authentication mechanisms, such as PIN codes or passcodes, that are not available to the forensic practitioner, or other encryption mechanisms that carry the risk of delaying or even blocking the acquisition of the mobile device. Removing the SIM card: In most mobile devices, this operation implies removing the battery and therefore all the risks and consequences we just mentioned regarding switching off the device; however, in iOS devices this task is quite straightforward and easy, and it does not imply removing the battery (in iOS devices this is not possible). Moreover, SIM cards can have PIN protection enabled; removing it from the phone may lock the SIM card, preventing its content from being displayed. However, bear in mind that removing the SIM card will isolate the device only from the cellular network, while other networks, such as Wi-Fi or Bluetooth, may still be active and therefore need to be addressed. The following image shows a SIM card extracted from an iPhone with just a clip; image taken from http://www.maclife.com/: Summary In this article, we gave a general introduction to digital forensics for those relatively new to this area of study and a good recap to those already in the field, keeping the mobile forensics field specifically in mind. We have shown what digital evidence is and how it should be handled, presenting several techniques to isolate the mobile device from the network. Resources for Article: Further resources on this subject: Mobile Forensics [article] Mobile Forensics and Its Challanges [article] Forensics Recovery [article]
Read more
  • 0
  • 0
  • 23716

article-image-ai-distilled-35-building-llms-rust-inference-speechgpt-gen-and-3d
Merlyn Shelley
02 Feb 2024
12 min read
Save for later

AI Distilled 35: Building LLMs, Rust Inference, SpeechGPT-Gen, and 3D

Merlyn Shelley
02 Feb 2024
12 min read
Dive deeper into the world of AI innovation and stay ahead of the AI curve! Subscribe to our AI_Distilled newsletter for the latest insights. Don't miss out – sign up today!👋 Hello ,“[AI healthcare systems] include understanding symptoms and conditions better, including simplifying explanations in local vernaculars…and acting as a valuable second opinion. This, in turn, potentially provides a pathway for medical AI towards superhuman diagnostic performance.” - Vivek Natrajan, Google AI Researcher AI is making rapid strides in healthcare with a major disruption underway. The recent study examined how current language models perform in diagnostic settings and it’s clear they’re making tremendous progress with each passing day. Step into AI_Distilled's latest installment, where we delve into the most recent developments in AI/ML, LLMs, NLP, GPT, and Gen AI. Join us as we kick off this edition, showcasing news and insights from various sectors: AI Industry Updates:  Apple Focuses on AI for Major iOS Update Tencent Chief Raises Concerns Over Gaming Business China Greenlights Over Four Dozen AI Systems New iPhone App Streamlines Web Searches with AI Assistance German Startup Aims to Supercharge AI Chips with Novel "Memcapacitor" Design AI Startup Anthropic Suffers Data Breach Amid Regulatory Scrutiny New AI Coding Tool Surpasses Previous Models New AI Model Launches:  Meta Unleashes Mighty Code Llama 70B Google Unveils New AI Model for Advanced Video Generation OpenAI Announces Major Model and API Updates AI in Healthcare: AI Helps Design Proteins for Improved Gene Therapy Delivery AI's Diagnostic Prowess Advancing Rapidly AI in Supply Chain Management: AI Transforming Global Supply Chain Management Edge AI's Potential in Logistics Faces Memory Limitations We’ve also got you your fresh dose of LLM, GPT, and Gen AI secret knowledge and tutorials: Assessing AI Accuracy: New Leaderboard Evaluates Language Models New Techniques to Boost AI Performance Neural Networks Learn Non-Linear Functions Through Rectified Activation Enhancing Conversations with Contextual Understanding We know how much you love hands-on tips and strategies from the community, so here they are: Fine-Tuning BERT for Long-Form Text Training Giant Language Models the Cost-Effective Way Optimizing AI Models for Speed and Scale Estimating Depth from a Single Image: Researchers Evaluate AI Models Don’t forget to review these GitHub repositories that have been doing rounds: rasbt/LLMs-from-scratch LaurentMazare/mamba.rs 0nutation/speechgpt naver/roma  📥 Feedback on the Weekly EditionTake our weekly survey and get a free PDF copy of our best-selling book, "Interactive Data Visualization with Python - Second Edition." 📣 And here's the twist – we're tuning into YOUR frequency! Inspired by a reader's request, we're launching a column just for you. Got a burning question or a topic you're itching to dive into? Drop your suggestions in our content box – because your journey of discovery is our blueprint.We appreciate your input and hope you enjoy the book! Share your thoughts and opinions here! Writer’s Credit: Special shout-out to Vidhu Jain for their valuable contribution to this week’s newsletter content!  Cheers,  Merlyn Shelley  Editor-in-Chief, Packt  🗝️ Unlock the Packt library for FREEDive into a world of endless knowledge with our 7-day FREE trial! Discover over 7,500 tech books and videos with your Packt subscription and stay ahead in your field.Plus, check out our ✨NEW feature: the AI Assistant (beta) ✨, available across eBook, print, and subscription formats.  Don't miss your chance to explore and innovate – start your free trial today and unlock your tech potential! Ready to Crush It? Start Upskilling! SignUp | Advertise | Archives⚡ TechWave: AI/GPT News & Analysis💎 Apple Focuses on AI for Major iOS Update: Apple is planning a major iOS 18 update, aiming to enhance AI features, particularly Siri and messaging apps. It will include generative AI in various apps, with Apple in talks with publishers for content. New iPads and MacBooks with M3 chips may be released soon. 💎 Tencent Chief Raises Concerns Over Gaming Business: Tencent's CEO, Pony Ma, is concerned about competition in the gaming division as recent titles underperformed while rivals thrive. Despite 30% of revenue from games, Tencent feels left behind. They've caught up in AI and plan to integrate it across businesses, with a focus on evolving WeChat platform. 💎 China Greenlights Over Four Dozen AI Systems: Chinese regulators have recently granted approval to over 40 AI models for public use, with companies including Baidu, Alibaba, and ByteDance receiving the green light. This move reflects China's push to narrow the AI gap with the United States, driven by the success of ChatGPT. 💎 New iPhone App Streamlines Web Searches with AI Assistance: Arc Search is a mobile app improving search with "Browse for Me" creating concise web pages from multiple sources, AI summaries, tab switching, and reading mode. It's an early-stage tool designed to save users time in the digital era. 💎 German Startup Aims to Supercharge AI Chips with Novel "Memcapacitor" Design: Semron, a German startup, aims to revolutionize the computer chip industry by replacing transistors with "memcapacitors." Their 3D-stacked memcapacitor design promises improved AI performance and reduced energy consumption. With $10 million in funding, Semron aims to provide efficient chips for mobile and edge AI applications. 💎 AI Startup Anthropic Suffers Data Breach Amid Regulatory Scrutiny: Anthropic, the creator of LLM, disclosed an accidental data leak by a contractor to a third party. Concurrently, the FTC initiated an inquiry into Anthropic's affiliations with Amazon and Google, fueling concerns about data privacy in the expanding LLM landscape. 💎 New AI Coding Tool Surpasses Previous Models: AlphaCodium, an open-source AI model by CodiumAI, surpasses Google's AlphaCode and AlphaCode 2 in code generation efficiency. It employs a "flow engineering" approach, emphasizing code integrity through iterative code generation and adversarial testing, aiming to advance upon DeepMind's work and assist global developers.  New AI Model Launches: 💎 Google Unveils New AI Model for Advanced Video Generation: Google has unveiled Lumiere, an advanced AI model utilizing space-time diffusion to create lifelike videos from text or images. It addresses motion and consistency problems in AI video generation, yielding 5-second clips with smooth motion. While not yet available for testing, experts anticipate Lumiere could redefine AI video creation capabilities. 💎 Meta Unleashes Mighty Code Llama 70B: Meta has launched Code Llama 70B, a powerful AI model for automating software development. With 500 billion training examples, it excels in translating natural language instructions into functional code, surpassing previous models. Meta aims to boost global coding and expand its use in translation, documentation, and debugging. 💎 OpenAI Announces Major Model and API Updates: OpenAI launched new AI models and tools, including enhanced embeddings, GPT-4, and moderation models. They lowered GPT-3.5 Turbo pricing, introduced key management features, and per-key analytics to enhance user control and accessibility, bolstering their technology's capabilities. AI in Healthcare: 💎 AI Helps Design Proteins for Improved Gene Therapy Delivery: University of Toronto scientists created ProteinVAE, an AI model to design unique protein variants, aiming to evade immune responses in gene therapy. By re-engineering adenovirus hexon proteins, they generate novel sequences for improved safety and efficacy, with faster, cost-effective design. Successful experiments could enhance gene therapy. 💎 AI's Diagnostic Prowess Advancing Rapidly: A study assessed GPT-3.5 and GPT-4's ability to mimic doctors' diagnostic reasoning. AI gave mostly accurate responses to various prompts, suggesting potential for assisting physicians, provided clinicians grasp the AI's response generation process.  AI in Supply Chain Management: 💎 AI Transforming Global Supply Chain Management: A recent study revealed that 98% of executives consider AI crucial for enhancing supply chain operations. AI aids in cost reduction through inventory optimization, transportation, and trade expense management. It particularly benefits inventory management, a vital cost-cutting aspect, as businesses increasingly adopt AI and data-driven solutions to overcome ongoing supply chain challenges and achieve strategic objectives like cost reduction and revenue growth. 💎 Edge AI's Potential in Logistics Faces Memory Limitations: AI and edge computing offer potential for efficient supply chain management through real-time decision-making. However, data processing at the edge strains memory. MRAM tech may alleviate limitations. As data increases, novel storage solutions are essential for maximizing edge AI's logistics benefits, hinging on memory improvements.  🔮 Expert Insights from Packt Community Complete Python Course with 10 Real-World Projects [Video] - By Ardit Sulce This Python course benefits both beginners and experienced AI developers by thoroughly covering Python's versatility in supporting different programming paradigms. It begins with a concise introduction and explores fundamental to advanced Python techniques. For beginners, the initial 12 sections provide a strong foundation in Python basics. Experienced developers can sharpen their skills by exploring intermediate and advanced concepts such as OOPS, classes, lists, modules, functions, and JSON. Additionally, the course extends into practical application by teaching the use of essential libraries like Matplotlib and NumPy, web development with Flask, and even Android APK file manipulation.  Database handling and geographical app development are also included, enriching the skill set of both novices and experts. The course culminates with the creation of ten practical applications, ranging from a volcano web map generator to data analysis dashboards, mobile apps, web scraping tools, and more. Ultimately, you will gain the ability to independently create executable Python programs, master coding syntax, and achieve comprehensive proficiency in Python programming, making this course highly recommended for students at all levels of experience. Plus, you can access project resources at: https://github.com/PacktPublishing/Complete-Python-Course-with-10-Real-World-Projects.Discover the "Complete Python Course with 10 Real-World Projects [Video]" by Ardit Sulce, published in February 2023. Get a comprehensive overview of the course content through a complete chapter preview video. Alternatively, access the entire Packt digital library, including this course and more, with a 7-day free trial. To access more insights and additional resources, click the button below.Watch Here 🌟 Secret Knowledge: AI/LLM Resources💎 Assessing AI Accuracy: New Leaderboard Evaluates Language Models: The Hallucinations Leaderboard assesses Language Models (LLMs) for generating incorrect information. It includes diverse tasks to test accuracy. Initial findings reveal variations in model performance, aiding in the pursuit of more reliable and less hallucination-prone LLMs. 💎 New Techniques to Boost AI Performance: Google researchers have created innovative software techniques for optimizing mixed-input matrix multiplication, a critical AI computation. They utilize register shuffling and efficient data type conversions to map mixed-input tasks onto hardware-accelerated Tensor Cores with minimal overhead. On NVIDIA GPUs, their approach matches the performance of hardware-native operations, offering potential solutions to computational challenges in advancing AI technology. 💎 Neural Networks Learn Non-Linear Functions Through Rectified Activation: The article explains how neural networks approximate complex functions, especially with Rectified Linear Unit (ReLU) activation. ReLU enables multiple neurons to represent linear and curved functions, emphasizing proper architecture over more layers for accuracy without overfitting, revealing neural networks' expressive capabilities. 💎 Enhancing Conversations with Contextual Understanding: The article explores enhancing conversational agents' contextual understanding in multi-turn conversations. It utilizes models like FLAN and Llama to assess question relationships, incorporates context from prior questions, and reevaluates responses to provide more comprehensive and successful conversation outcomes. These methods showed promising results in testing. 🔛 Masterclass: AI/LLM Tutorials💎 Fine-Tuning BERT for Long-Form Text: This article discusses how to use powerful NLP models like BERT to analyze long passages. BERT works by splitting lengthy reviews into chunks that fit its 512-token limit. It first pre-trains on general text, then fine-tunes on a task. To classify a long movie review, it is chopped into pieces, tagged with IDs, and stacked into the model. Results are pooled to get an overall sentiment.  💎 Training Giant Language Models the Cost-Effective Way: This guide explains how to enhance AI model training on AWS using Trainium and EKS. Set up a Kubernetes cluster with Trainium chips, preprocess data with Llama2's tokenizer, and optimize the model with compilation jobs. Monitor training across nodes, checking logs, utilization, and using Tensorboard for cost-effective training of large models. 💎 Optimizing AI Models for Speed and Scale: This blog advises on efficiently setting up and optimizing large language models for speed and scalability. It emphasizes hardware and configuration choices tailored to each model's requirements, balancing factors like response time and concurrent user capacity. Benchmarks demonstrate how GPU usage, model distribution, and request volume affect performance. Proper tuning enables AI assistants to engage in concurrent conversations with many users without excessive delays. 💎 Estimating Depth from a Single Image: Researchers Evaluate AI Models: Researchers evaluated neural networks' monocular depth estimation capabilities using single photos and RGB-depth map datasets. DPT and Marigold models were trained and compared, with DPT outperforming in accuracy based on RMSE and SSIM metrics. While promising, further enhancements are needed in this area.  🚀 HackHub: Trending AI Tools💎 rasbt/LLMs-from-scratch: Incrementally build a ChatGPT-like LLM from scratch for educational purposes.💎 LaurentMazare/mamba.rs: Pure Rust version of the Mamba inference algorithm with minimal dependencies for efficient sequence modeling. 💎 0nutation/speechgpt: Allows multi-modal conversations and SpeechGPT-Gen for scaling chain-of-information speech generation.💎 naver/roma: PyTorch library offering useful tools for working with 3D rotations, including conversions between rotation representations, metrics, interpolation, and differentiation capabilities.
Read more
  • 0
  • 0
  • 23709
article-image-java-refactoring-netbeans
Packt
08 Jun 2011
7 min read
Save for later

Java Refactoring in NetBeans

Packt
08 Jun 2011
7 min read
  NetBeans IDE 7 Cookbook Over 70 highly focused practical recipes to maximize your output with NetBeans         Introduction Be warned that many of the refactoring techniques presented in this article might break some code. NetBeans, and other IDEs for that matter too, make it easier to revert changes but of course be wary of things going wrong. With that in mind, let's dig in. Renaming elements This recipe focuses on how the IDE handles the renaming of all elements of a project, being the project itself, classes, methods, variables, and packages. How to do it... Let's create the code to be renamed: Create a new project, this can be achieved by either clicking File and then New Project or pressing Ctrl+Shift+N. On New Project window, choose Java on the Categories side, and on the Projects side select Java Application. Then click Next. Under Name and Location: name the project as RenameElements and click Finish. With the project created we will need to clear the RenameElements.java class of the main method and insert the following code: package renameelements; import java.io.File; public class RenameElements { private void printFiles(String string) { File file = new File(string); if (file.isFile()) { System.out.println(file.getPath()); } else if (file.isDirectory()) { for(String directory : file.list()) printFiles(string + file.separator + directory); } if (!file.exists()) System.out.println(string + " does not exist."); } } The next step is to rename the package, so place the cursor on top of the package name, renameelements, and press Ctrl+R. A Rename dialog pops-up with the package name. Type util under New Name and click on Refactor. Our class contains several variables we can rename: Place the cursor on the top of the String parameter named string and press Ctrl+R. Type path and press Enter Let's rename the other variables: Rename file into filePath. To rename methods, perform the steps below: Place the cursor on the top of the method declaration, printFiles, right-click it then select Refactor and Rename.... On the Rename Method dialog, under New Name enter recursiveFilePrinting and press Refactor. Then let's rename classes: To rename a class navigate to the Projects window and press Ctrl+R on the RenameElements.java file. On the Rename Class dialog enter FileManipulator and press Enter. And finally renaming an entire project: Navigate to the Project window, right-click on the project name, RenamingElements, and choose Rename.... Under Project Name enter FileSystem and tick Also Rename Project Folder; after that, click on Rename. How it works... Renaming a project works a bit differently from renaming a variable, since in this action NetBeans needs to rename the folder where the project is placed. The Ctrl+R shortcut is not enough in itself so NetBeans shows the Rename Project dialog. This emphasizes to the developer that something deeper is happening. When renaming a project, NetBeans gives the developer the possibility of renaming the folder where the project is contained to the same name of the project. This is a good practice and, more often than not, is followed. Moving elements NetBeans enables the developer to easily move classes around different projects and packages. No more breaking compatibility when moving those classes around, since all are seamlessly handled by the IDE. Getting ready For this recipe we will need a Java project and a Java class so we can exemplify how moving elements really work. The exisiting code, created in the previous recipe, is going to be enough. Also you can try doing this with your own code since moving classes are not such a complicated step that can't be undone. Let's create a project: Create a new project, which can be achieved either by clicking File and then New Project or pressing Ctrl+Shift+N. In the New Project window, choose Java on the Categories side and Java Application on the Projects side, then click Next. Under Name and Location, name the Project as MovingElements and click Finish. Now right-click on the movingelements package, select New... and Java Class.... On the New Java Class dialog enter the class name as Person. Leave all the other fields with their default values and click Finish. How to do it... Place the cursor inside of Person.java and press Ctrl+M. Select a working project from Project field. Select Source Packages in the Location field. Under the To Package field enter: classextraction: How it works... When clicking the Refactor button the class is removed from the current project and placed in the project that was selected from the dialog. The package in that class is then updated to match. Extracting a superclass Extracting superclasses enables NetBeans to add different levels of hierarchy even after the code is written. Usually, requirements changing in the middle of development, and rewriting classes to support inheritance would quite complicated and time-consuming. NetBeans enables the developer to create those superclasses in a few clicks and, by understanding how this mechanism works, even creates superclasses that extend other superclasses. Getting ready We will need to create a Project based on the Getting Ready section of the previous recipe, since it is very similar. The only change from the previous recipe is that this recipe's project name will be SuperClassExtraction. After project creation: Right-click on the superclassextraction package, select New... and Java Class.... On the New Java Class dialog enter the class name as DataAnalyzer. Leave all the other fields with their default values and click Finish. Replace the entire content of the DataAnalyzer.java with the following code: package superclassextraction; import java.util.ArrayList; public class DataAnalyzer { ArrayList<String> data; static final boolean CORRECT = true; static final boolean INCORRECT = false; private void fetchData() { //code } void saveData() { } public boolean parseData() { return CORRECT; } public String analyzeData(ArrayList<String> data, int offset) { //code return ""; } } Now let's extract our superclass. How to do it... Right-click inside of the DataAnalyzer.java class, select Refactor and Extract Superclass.... When the Extract Superclass dialog appears, enter Superclass Name as Analyzer. On Members to Extract, select all members, but leave saveData out. Under the Make Abstract column select analyzeData() and leave parseData(), saveData(), fetchData() out. Then click Refactor. How it works... When the Refactor button is pressed, NetBeans copies the marked methods from DataAnalyzer.java and re-creates them in the superclass. NetBeans deals intelligently with methods marked as abstract. The abstract methods are moved up in the hierarchy and the implementation is left in the concrete class. In our example analyzeData is moved to the abstract class but marked as abstract; the real implementation is then left in DataAnalyzer. NetBeans also supports the moving of fields, in our case the CORRECT and INCORRECT fields. The following is the code in DataAnalyzer.java: public class DataAnalyzer extends Analyzer { public void saveData() { //code } public String analyzeData(ArrayList<String> data, int offset) { //code return ""; } } The following is the code in Analyzer.java: public abstract class Analyzer { static final boolean CORRECT = true; static final boolean INCORRECT = false; ArrayList<String> data; public Analyzer() { } public abstract String analyzeData(ArrayList<String> data, int offset); public void fetchData() { //code } public boolean parseData() { //code return DataAnalyzer.CORRECT; } } There's more... Let's learn how to implement parent class methods. Implementing parent class methods Let's add a method to the parent class: Open Analyzer.java and enter the following code: public void clearData(){ data.clear(); } Save the file. Open DataAnalyzer.java, press Alt+Insert and select Override Method.... In the Generate Override Methods dialog select the clearData() option and click Generate. NetBeans will then override the method and add the implementation to DataAnalyzer.java: @Override public void clearData() { super.clearData(); }  
Read more
  • 0
  • 0
  • 23700

article-image-introduction-hlsl-language
Packt
28 Jun 2013
8 min read
Save for later

Introduction to HLSL language

Packt
28 Jun 2013
8 min read
(For more resources related to this topic, see here.) Distance/Height-based fog Distance/Height-based fog is an approximation to the fog you would normally see outdoors. Even in the clearest of days, you should be able to see some fog far in the distance. The main benefit of adding the fog effect is that it helps the viewer estimate how far different elements in the scene are based on the amount of fog covering them. In addition to the realism this effect adds, it has the additional benefit of hiding the end of the visible range. Without fog to cover the far plane, it becomes easier to notice when far scene elements are clipped by the cameras far plane. By tuning the height of the fog you can also add a darker atmosphere to your scene as demonstrated by the following image: This recipe will demonstrate how distance/height-based fog can be added to our deferred directional light calculation. See the How it works… section for details about adding the effect to other elements of your rendering code. Getting ready We will be passing additional fog specific parameters to the directional light's pixel shader through a new constant buffer. The reason for separating the fog values into their own constant buffer is to allow the same parameters to be used by any other shader that takes fog into account. To create the new constant buffer use the following buffer descriptor: Constant buffer descriptor parameter   Value   Usage   D3D11_USAGE_DYNAMIC   BindFlags   D3D11_BIND_CONSTANT_BUFFER   CPUAccessFlags   D3D11_CPU_ACCESS_WRITE   ByteWidth   48   The reset of the descriptor fields should be set to zero. All the fog calculations will be handled in the deferred directional light pixel shader. How to do it... Our new fog constant buffer is declared in the pixel shader as follows: cbuffer cbFog : register( b2 ){float3 FogColor : packoffset( c0 );float FogStartDepth : packoffset( c0.w );float3 FogHighlightColor : packoffset( c1 );float FogGlobalDensity : packoffset( c1.w );float3 FogSunDir : packoffset( c2 );FogHeightFalloff : packoffset( c2.w );} The helper function used for calculating the fog is as follows: float3 ApplyFog(float3 originalColor, float eyePosY, float3eyeToPixel){float pixelDist = length( eyeToPixel );float3 eyeToPixelNorm = eyeToPixel / pixelDist;// Find the fog staring distance to pixel distancefloat fogDist = max(pixelDist - FogStartDist, 0.0);// Distance based fog intensityfloat fogHeightDensityAtViewer = exp( -FogHeightFalloff * eyePosY );float fogDistInt = fogDist * fogHeightDensityAtViewer;// Height based fog intensityfloat eyeToPixelY = eyeToPixel.y * ( fogDist / pixelDist );float t = FogHeightFalloff * eyeToPixelY;const float thresholdT = 0.01;float fogHeightInt = abs( t ) > thresholdT ?( 1.0 - exp( -t ) ) / t : 1.0;// Combine both factors to get the final factorfloat fogFinalFactor = exp( -FogGlobalDensity * fogDistInt *fogHeightInt );// Find the sun highlight and use it to blend the fog colorfloat sunHighlightFactor = saturate(dot(eyeToPixelNorm, FogSunDir));sunHighlightFactor = pow(sunHighlightFactor, 8.0);float3 fogFinalColor = lerp(FogColor, FogHighlightColor,sunHighlightFactor);return lerp(fogFinalColor, originalColor, fogFinalFactor);} The Applyfog function takes the color without fog along with the camera height and the vector from the camera to the pixel the color belongs to and returns the pixel color with fog. To add fog to the deferred directional light, change the directional entry point to the following code: float4 DirLightPS(VS_OUTPUT In) : SV_TARGET{// Unpack the GBufferfloat2 uv = In.Position.xy;//In.UV.xy;SURFACE_DATA gbd = UnpackGBuffer_Loc(int3(uv, 0));// Convert the data into the material structureMaterial mat;MaterialFromGBuffer(gbd, mat);// Reconstruct the world positionfloat2 cpPos = In.UV.xy * float2(2.0, -2.0) - float2(1.0, -1.0);float3 position = CalcWorldPos(cpPos, gbd.LinearDepth);// Get the AO valuefloat ao = AOTexture.Sample(LinearSampler, In.UV);// Calculate the light contributionfloat4 finalColor;finalColor.xyz = CalcAmbient(mat.normal, mat.diffuseColor.xyz) * ao;finalColor.xyz += CalcDirectional(position, mat);finalColor.w = 1.0;// Apply the fog to the final colorfloat3 eyeToPixel = position - EyePosition;finalColor.xyz = ApplyFog(finalColor.xyz, EyePosition.y,eyeToPixel);return finalColor;} With this change, we apply the fog on top of the lit pixels color and return it to the light accumulation buffer. How it works… Fog is probably the first volumetric effect implemented using a programmable pixel shader as those became commonly supported by GPUs. Originally, fog was implemented in hardware (fixed pipeline) and only took distance into account. As GPUs became more powerful, the hardware distance based fog was replaced by a programmable version that also took into account things such as height and sun effect. In reality, fog is just particles in the air that absorb and reflect light. A ray of light traveling from a position in the scene travels, the camera interacts with the fog particles, and gets changed based on those interactions. The further this ray has to travel before it reaches the camera, the larger the chance is that this ray will get either partially or fully absorbed. In addition to absorption, a ray traveling in a different direction may get reflected towards the camera and add to the intensity of the original ray. Based on the amount of particles in the air and the distance a ray has to travel, the light reaching our camera may contain more reflection and less of the original ray which leads to a homogenous color we perceive as fog. The parameters used in the fog calculation are: FogColor: The fog base color (this color's brightness should match the overall intensity so it won't get blown by the bloom) FogStartDistance: The distance from the camera at which the fog starts to blend in FogHighlightColor: The color used for highlighting pixels with pixel to camera vector that is close to parallel with the camera to sun vector FogGlobalDensity: Density factor for the fog (the higher this is the denser the fog will be) FogSunDir: Normalized sun direction FogHeightFalloff: Height falloff value (the higher this value, the lower is the height at which the fog disappears will be) When tuning the fog values, make sure the ambient colors match the fog. This type of fog is designed for outdoor environments, so you should probably disable it when lighting interiors. You may have noticed that the fog requires the sun direction. We already store the inversed sun direction for the directional light calculation. You can remove that value from the directional light constant buffer and use the fog vector instead to avoid the duplicate values This recipe implements the fog using the exponential function. The reason for using the exponent function is because of its asymptote on the negative side of its graph. Our fog implementation uses that asymptote to blend the fog in from the starting distances. As a reminder, the exponent function graph is as follows: The ApplyFog function starts off by finding the distance our ray traveled in the fog (fogDepth). In order to take the fog's height into account, we also look for the lowest height between the camera and the pixel we apply the fog to which we then use to find how far our ray travels vertically inside the fog (fogHeight). Both distance values are negated and multiplied by the fog density to be used as the exponent. The reason we negate the distance values is because it's more convenient to use the negative side of the exponential functions graph which is limited to the range 0 to 1. As the function equals 1 when the exponent is 0, we have to invert the results (stored in fogFactors). At this point we have one factor for the height which gets larger the further the ray travels vertically into the fog and a factor that gets larger the further the ray travels in the fog in any direction. By multiplying both factors with each other we get the combined fog effect on the ray: the higher the result is, the more the original ray got absorbed and light got reflected towards the camera in its direction (this is stored in fogFinalFactor). Before we can compute the final color value, we need to find the fog's color based on the camera and sun direction. We assume that the sun intensity is high enough to get more of its light rays reflected towards the camera direction and sun direction are close to parallel. We use the dot product between the two to determine the angle and narrow the result by raising it to the power of 8 (the result is stored in sunHighlightFactor). The result is used to lerp between the fog base color and the fog color highlighted by the sun. Finally, we use the fog factor to linearly interpolate between the input color and the fog color. The resulting color is then returned from the helper function and stored into the light accumulation buffer. As you can see, the changes to the directional light entry point are very minor as most of the work is handled inside the helper function ApplyFog. Adding the fog calculation to the rest of the deferred and forward light sources should be pretty straightforward. One thing to take into consideration is that fog also has to be applied to scene elements that don't get lit, like the sky or emissive elements. Again, all you have to do is call ApplyFog to get the final color with the fog effect. Summary In this article, we learned how to apply fog effect and add atmospheric scenes to the images. Resources for Article : Further resources on this subject: Creating and Warping 3D Text with Away3D 3.6 [Article] 3D Vector Drawing and Text with Papervision3D: Part 1 [Article] 3D Vector Drawing and Text with Papervision3D: Part 2 [Article]
Read more
  • 0
  • 0
  • 23689
Modal Close icon
Modal Close icon